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For the thirty-ninth time, the Research and Theory Division of the Association for Educational 
Communications and Technology (AECT) is sponsoring the publication of these Proceedings. Papers 
published in this volume were presented at the annual AECT Convention in Jacksonville. FL. A limited 
quantity of these Proceedings were printed and sold in both hardcopy and electronic versions. Volumes | 
and 2 are available through the Educational Resources Clearinghouse (ERIC) System. Proceedings volumes 
are available to members at AECT.ORG. Proceedings copies are also available at: 


http://www.tresystems.com/proceedings/ 


The Proceedings of AECT’s Convention are published in two volumes. Volume #1 contains papers dealing 
primarily with research and development topics. Papers dealing with the practice of instructional 
technology including instruction and training issues are contained in Volume #2. This year, both volumes 
are included in one document. 


REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings. The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 
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Gamification for Change: A New Approach to Investigate Students’ Attitudes 
Toward Educational Gamification in Online Learning Environments 


Sumayah Abu-Dawood 
University of North Texas 
sabudawood@gmail.com 


Introduction 


Gamification is “the use of game design elements in non-game contexts” (Deterding, Dixon, Khaled, & 
Nacke, 2011, p. 10). Gamification in education differs from creating a full-fledged game as it focuses on extracting 
the factors that make video games fun and pleasurable and adapts those factors into course design (Deterding et al., 
2011; Measles & Abu-Dawood, 2015). The purpose of educational gamification is to motivate students by creating 
an engaging learning experience that can keep students focused on the learning task (Abudawood, Measles, 
Madrigal, & Kaplan, 2015). Instructional designers may insert one or more game design elements like: (a) badges, 
(b) leaderboards, (c) points, (d) levels, (e) storylines, or (f) avatars to promote students’ interactions with each other 
and their interactions with the course content. 

In education, most empirical gamification studies focus on the effectiveness of using game design elements 
in specific learning contexts (Dicheva, Dichev, Agre, & Angelova, 2015). Unfortunately, researching students’ 
attitudes towards gamification in online learning environments is limited. In addition, there is a lack of “validated 
psychometric measurements” created according to rigorous research methodologies (Ortiz, Chiluiza, & Valcke, 
2016). Most of the gamification research in higher education that sought students’ perspectives was conducted after 
students’ involvement in a gamified learning experience (Chou & He, 2017; Denny, 2013; Dominguez et al., 2013; 
Fotaris, Mastoras, Leinfellner, & Rosunally, 2016; Kumar & Khurana, 2012; O’Donovan, Gain, & Marais, 2013; 
Pettit, McCoy, Kinney, & Schwartz, 2015). Because developing gamified online courses are costly endeavors 
(Bernik, Bubas, & Radosevic, 2015), using validated instruments to investigate students’ attitudes toward 
gamification in online learning environment before implementation is needed. 

To overcome the lack of rigorous methodologies that investigated students’ attitudes before the 
implementation of educational gamification, this paper reports a research methodology to investigate graduate and 
undergraduate students’ attitudes toward gamification in online learning environments. This explanatory mixed- 
method approach employs both quantitative and qualitative methods to explore students’ perspectives toward the 
pleasurability of different learning experiences. These learning experiences could be gamified and implemented in 
online courses using various game elements: (a) feedback, (b) points, (c) progress bar, (d) storytelling, (e) badges, 
and (f) levels. The current approach is based on: (a) the concept of pleasurability, (b) the self-determination theory 
of motivation (SDT), and (c) situated learning. This approach applies two methods for investigation: (1) Pleasurable 
Learning Experiences Scale (PLLEXs) (A, 2016) to assess students’ attitudes toward video games and gamification 
quantitatively and (2) semistructured interviews to assess students’ attitudes toward gamification qualitatively. 

This new approach proposes to identify the main key features of educational gamification from students’ 
perspectives with the objective of changing the way of identifying students’ needs before the implementation of 
educational gamification in online learning environments. This change in approach will inform our practices as 
instructors and instructional designers in the field of learning technologies to enhance students’ learning and 
performance in higher educational institutions. 


Theoretical Background of Educational Gamification 
The Concept of Pleasurability 


The concept of pleasurability could be considered from two perspectives: (a) Human-computer Interaction 
(HCI) design and (b) video game design. From an HCI design perspective, the effective product design should meet 
three consumer needs proposed by Jordan (2003) as a three-level hierarchy: (1) functionality: refers to the product 
containing the necessary functions to perform the intended tasks, (2) usability: refers to the product being easy to 
use, (3) pleasurability: refers to the product being a living interactive object that brings emotional joy when used. 


The pleasurability of the product can be further classified into four types: (a) physio pleasure, (b) socio pleasure, (c) 
psycho pleasure, and (d) ideo pleasure (A, 2016; Jordan, 2003). These types of pleasure are defined as 


physio pleasure—the pleasure derived from our senses such as touch and feel, 

socio pleasure—the pleasure derived from having social relationship with others, 

psycho pleasure—the satisfaction derived from the feeling of competence when having the 
required skills and resources to accomplish a certain task, and 

ideo pleasure—the emotional pleasure derived from people’s values of a specific theoretical 
entity, like the emotional value of using environmental friendly products that goes beyond the 
functionality and usability of the product (A, 2016; Jordan, 2003). 


These four types of pleasure can be theorized from video game design perspectives to inform the design of 
educational gamification by utilizing various game elements that can evoke different pleasurable aspects of learning 


(A, 2016). Table 1| illustrates the four pleasurability aspects in video games using game elements. 


Table 1. Four Pleasurable Aspects of Video Game Elements 


Physio Pleasure Socio Pleasure Psycho Pleasure Ideo Pleasure 
Points Collaboration Challenges Badges 
: Leaderboards 
Progress Tracking Feedback Levels : 
Storytelling 


Note. A (2016) 


Educational gamification aims to motivate and engage students in their learning. Designing gamified 
courses should make learning more pleasurable and fun (A, 2016). The Playful Experiences (PLEX) framework is a 
game design framework that classifies the fun and playful factors that one would experience while interacting with 
video games into 22 categories (Arrasvuori, Boberg, Holopainen, Korhonen, Lucero, & Montola, 2011). A (2016) 
found that only seven categories from the 22 PLEX categories were appropriate for use in learning contexts with 
two additional categories added (interaction and visualization). These nine categories formed the Pleasurable 
Learning Experiences Scale (PLLEXs), as illustrated in Figure 1. 


Control 


Dominating, commanding, 
regulating 


Sensation 


Excitement by stimulating 
senses 


Challenge 


Testing abilities in a 
demanding task 


Submission 


Being part of a larger 
structure 


Fellowship 


Friendship, communality, or 
intimacy 


Interaction 


A reciprocal action, effect, 
or influence 


Humor 


Fun, joy, amusement, jokes, 


gags 


Discovery 


Finding something new or 


unknown 


Visualization 


Recalling or forming mental 


images 


Figure 1. The Nine Categories of the Pleasurable Learning Experiences Scale (PLLEXs) 


Self-Determination Theory of Motivation 


Human motivation is vital to understanding why people are moved to do a certain task. There are two 
sources of human motivation that can drive someone to do a specific task: (a) the task itself being interesting and 
enjoyable, which refers to the task features, and (b) the satisfaction that individuals gain from performing a task, 
which refers to the satisfaction of innate psychological needs (Ryan & Deci, 2000). The self-determination theory 
(SDT) is a macro theory that focuses on understanding the innate psychological needs that spur a student’s 
motivation (Ryan & Deci, 2000). In this theory, Ryan and Deci distinguished between two kinds of motivation: 
intrinsic motivation and extrinsic motivation (2000). When the task is intrinsically motivating, the person continues 
their involvement due to the feeling of enjoyment, rather than from attaining external rewards. According to SDT, a 
task becomes intrinsically motivating when three basic psychological needs are satisfied (Ryan, Rigby, & 
Przybylski, 2006): 


the need for autonomy, which means the task is done because of its personal value; 

the need for competence, which means the task is difficult but doable and doing it can enhance the 
feeling of self-efficacy; and 

the need for relatedness, which means the task can afford social connection (Ryan et al., 2006). 


While intrinsic motivation is an independent construct from extrinsic motivation, they do not oppose each 
other because human motivation can also be theorized as a continuum from fully internal to fully external 
motivation (Ryan & Deci, 2000). Therefore, the balance between intrinsic and extrinsic motivators in designing 
gamified online learning environments is essential to sustaining student engagement. 

Engagement focuses on sustaining student attention by creating an immersive learning environment that 
can motivate students to continue working on the learning task. SDT has been utilized in various contexts, including 
video game play, to understand how games work to satisfy players’ psychological needs (Przybylski, Rigby, & 
Ryan, 2010). Playing video games can be an intrinsically motivating activity because players may feel competent, 
autonomous, and related as a result of their video game play (Ryan et al., 2006). These three needs have been found 
to be strong predictors of game enjoyment and of future game play (Ryan et al., 2006). In light of SDT, gamification 
in online learning environments should address these three innate psychological needs in order to maintain students’ 
intrinsic motivation by providing (a) freedom of choice to allow for autonomy, (b) challenging but doable tasks to 
promote competence, and (c) communication and collaboration opportunities for social connection (Aparicio, Vela, 
Sanchez, & Montes, 2012). 

While PLLEXs is based on the PLEX framework, considering the underlying psychological needs that can 
stimulate students’ intrinsic motivation is important. Based on these three innate psychological needs—autonomy, 
competence, and relatedness—the proposed approach connects the PLLEXs subscales and categories to the 
associated psychological needs that should be satisfied to motivate students in gamified online learning 
environments, as illustrated in Table 2. 


Table 2. PLLEXs Subscales and Categories with Associated Psychological Needs According to SDT 


PLLEXs PLLEXs : : 
ee caas Catepories Associated SDT Psychological Needs 
Preferences for Control Autonomy: The gamified course provides freedom of choice to 
Instructions promote the feeling of the learner being in control of his/her 
learning path. 
Submission Relatedness: The gamified course provides opportunities for 
learners to feel that their work is one part of a larger structure. 
Preferences for Humor Relatedness: The gamified course is designed in a way that 
Teaching Styles reflects a sense of humor to promote the feeling of fun. 
Sensation Competence: The gamified course promotes the feeling of 


competence by stimulating students’ senses through the use of 
multimedia (video) to illustrate skills. 


Petes ane Associated SDT Psychological Needs 


Subscales Categories 
Visualization Competence: The gamified course promotes the feeling of 
competence by using multimedia (video) to help students recall 
information or form a mental image of a concept or a skill. 
Preference for Challenge Competence: The gamified course promotes the feeling of 
Activities competence by providing learning tasks that are challenging but 
doable. 
Discovery Autonomy: The gamified course provides various opportunities 
for students to try new things. 
Fellowship Relatedness: The gamified course provides opportunities for 
interaction and communication to promote the feeling of 
fellowship. 
Interaction Relatedness: The gamified course provides opportunities for 
reciprocal action to promote interaction. 
Preferences for Fellowship Relatedness: The gamified course provides opportunities to 
Learning build rapport. 
Effectiveness 
Control Autonomy: The gamified course provides opportunities to 


control the flow of learning content. 
Note: PLLEXs = Pleasurable Learning Experiences Scale (A, 2016); SDT = self-determination theory 
(Ryan & Deci, 2000) 


Situated Learning 


Situated learning considers learning that occurs “in the same context in which it is applied, constructed 
socially by participants through their interactions and collaborations” (Abu-Dawood, 2016, p. 374). As a term, 
situated learning was invented by Lave and Wenger in 1991, when they described learning as “a process that takes 
place in a participation framework, not in an individual mind” (p. 15). According to this theory, learning is situated 
in an authentic context that encompasses knowledge and skills to be acquired. Through participation in such 
authentic contexts, learners become involved in a community of practice; this community embodies unique identities 
and behaviors and when learners acquire those behaviors, novice learners become experts as they move from the 
periphery of that community to its center (Lave & Wenger, 1991). 

In video games, players learn how to play the game itself through rapid cycles of expertise, which means 
that players are involved in repeated cycles of practicing skills until these skills become nearly automated, then 
having these skills challenged in a new situation, which necessitates acquiring a new set of skills and continued 
practicing until reaching a higher level of mastery (Gee, 2004). This continuous involvement in the rapid cycles of 
expertise enables players to try different identities and transfer their gained experiences into new situations. 
Essentially, games are socially and culturally situated systems, which means that games are one component in 
complex social systems that include players as actors and social norms as resources that can enable or disable the 
course of play (Ramirez & Squire, 2014). This notion has implications for designing gamified online learning 
environments. 

Educational gamification can provide opportunities for learners to take new roles and try new identities, 
such as doctors, lawyers, writers, or instructional designers. Because learning is situated in an authentic context, 
gamified online courses should provide opportunities for students to be involved in rapid cycles of expertise that 
shape failure as a necessary step to succeed until reaching level of mastery. The use of storyline can create learning 
situations where students can construct their experience through cognitive course content interactions and social 
interactions with others (Abu-Dawood, 2016). Gee (2008) discussed the projective identity in games using a situated 
learning matrix, in which “learning moves from identity to goals and norms, to tools and technologies, and only then 


to content” (p. 37). This matrix signifies the necessary conditions to be met in order to construct these experiences; 
these conditions are not limited to individuals’ cognition but also consider the active participation in a social system 
as part of meaning making (Gee, 2008). The design of educational gamification should consider rethinking the roles 
of students and the assessment mechanism used to promote students’ competence and encourage them to try new 
identities and to have a wide range of experiences to create shared knowledge. 

In this proposed approach, the use of storytelling as a game element is proposed and supported by situating 
learning theory. Students’ perspectives toward the pleasurability of different learning experiences can be 
investigated where storytelling as a game element is mapped to more than one learning experience in PLLEXs. The 
use of storytelling in PLLEXs serves three goals: (a) as rules given to the learners, (b) as narratives used to present 
the learning content, and (c) as a scaffolding mechanism to provide supportive information as needed. 


A New Approach for Investigating Students’ Attitudes Toward Educational Gamification 


The proposed new approach to investigate students’ attitudes toward the pleasurability of educational 
gamification in online learning environments is based on explanatory sequential design, discussed by Creswell and 
Plano Clark (2011). This two-phase approach supports the use of both qualitative and quantitative methods for 
investigation. The first phase focuses on collecting quantitative data using PLLEXs to explore the pleasurability of 
22 learning experiences from students’ perspectives. The second phase focuses on collecting qualitative data by 
conducting semistructured interviews with selected students to get their insights regarding their rationales of 
considering certain learning experiences as pleasurable or unpleasurable ones. According to this approach, the 
collected data is integrated during the interpretation stage, as illustrated in Figure 2. The analysis for each method 
should be carried out independently; then, the quantitative results would be further explained by the qualitative 
findings. In this way, the statistical quantitative results are compared with the qualitative thematic findings to search 
for possible additional themes that might emerge. The quantitative phase informs the plan and the implementation of 
the qualitative phase. The results should be reported giving detailed description of participants’ experiences and the 
researchers’ interpretations to provide rich contextualized information to the reader. Both quantitative and 
qualitative methods will be discussed in detail. 


Quantitative Qualitative 


Quantitative —» Quantitative —» Quantitative —> Purposeful Sample 
Data Collection Data Analysis Results 

PLLEXs: PLLEXs Qualitative Data 

- Demographics Collection Interviews 
- Online Exp. 


: aa Qualitative Data Analysis 
- Attitudes | 


Qualitative Findings 


Y 


Interpretation <——_—__ Datta Results Compared 


Figure 2: The Approach Design for Investigating Students' Attitudes Toward Educational Gamification 
Pleasurable Learning Experiences Scale (PLLEXs) 


As this approach encompasses both quantitative and qualitative phases, the quantitative phase employs a 
nonexperimental one-time attitudinal survey (PLLEXs). The PLLEXs is a 4-point Likert scale created by A (2016) 
to fulfill the need for having a quantifiable measure to assess students’ attitudes toward the pleasurability of different 
learning experiences. These learning experiences can be gamified using specific game elements, thus creating 
educational gamification. The PLLEXs is based on the Playful Experiences (PLEX) framework (Arrasvuori et al., 


2011). The PLEX framework encompasses 22 categories that describe the playfulness aspects that a user can 
experience while interacting with video games (Arrasvuori et al., 2011). However, PLLEXs consists of seven 
categories out of the 22, with two additional categories that emerged during the application of the scale in learning 
contexts (A, 2016), for a total of nine categories. According to A (2016), these nine categories demonstrate the 
pleasurable feelings that college students may experience in their learning environments. These categories were 


illustrated in Figure 1. 


Due to the nonexperimental nature of this approach, the use of PLLEXs is not intended to test hypotheses 
but to gain useful insights regarding students’ perspectives toward gamification and establish priorities as the 
research proceeds. PLLEXs includes two types of variables: (a) independent variables and (b) dependent variables. 
The independent variables occupy the first three parts of the survey: (a) demographics, (b) online learning 
experience, and (c) gaming experience. The questions in these three parts are designed to serve two goals: (a) 
describing the participants and (b) looking for possible correlations between participants’ attributes and their 
attitudes. Because gamification is stemmed from video games, understanding students’ gaming habits is essential to 
explore any possible correlation between video games experience and students’ attitudes toward educational 
gamification. The descriptions of all independent variables that are related to students experience with video games 


are illustrated in Table 3. 


Table 3. Description of Gaming Experience Independent Variables 


Independent Variables 


Previous Gaming 
Experience 
Current Gaming 
Experience 
Gaming Frequency 


Numbers of Gaming Hours 
Gaming Motivation 

Level of Gaming 
Experience 

Game Genre 

Player Type 


Gaming Likeness 


Description 


Students self-report whether they have played video games before. 


Students self-report the number of video games they are currently playing: 0, 1-2, 
3-4, 5-7, more than 7 games. 

Students self-report how frequently they play video games: daily, weekly, monthly, 
rarely, or never. 

Students self-report the number of hours they spend every week playing video 
games: 0, 1-3, 4-6, or more than 6 hours. 

Students self-report their reason for playing video games: to play with friends, to 
make new friends, as a mental challenge, as a physical challenge, to relieve stress, 
to get rid of boredom, none, or other (specify). 

Students self-report their level of experience in playing video games: no 
experience, novice, intermediate, or expert. 

Students self-report their favorite types of video games: puzzle, casual, strategy, 
action, adventure, fighting, shooter, racing, sports, role-playing, MMOVGs, or 
none. 

Students self-report their type as players: explorer, achiever, winner, socializer, or 
none. 

Students self-report how much they like playing video games by dragging an 
indicator on a scale (0-10). 


Note. MMOVGs = Massively Multiplayer Online Video Games. 


The fourth part of the survey is a 4-point Likert scale that encompasses 22 items. These items are organized 
into four subscales, which constitute the four dependent variables: 


Preferences for Instructions (8 items), 
Preferences for Instructors (3 items), 

Attitudes towards Activities (6 items), and 
Preferences for Learning Effectiveness (5 items). 


Each item represents a specific learning experience. Students can indicate their agreement/disagreement 
with the pleasurability of each learning experience by choosing one answer from the four possible choices: (1) 
strongly disagree, (2) disagree, (3) agree, and (4) strongly agree. The more agreement the student shows on any 
item, the more weight will be assigned to that item. For example, if a student responds to the item “T like instructors 


to provide clear rubrics.” by selecting “strongly agree,” the weight that will be given to that item is 4 and it could be 
inferred that the student perceived this experience as a fun and playful learning experience. 


Semistructured Interviews 


Semistructured interviews are the second phase of this approach and will be conducted to provide rich 
details on students’ perspectives towards gamification in online learning environments. This qualitative phase of the 
current approach will help to further explain the correlations that might exist between dependent and independent 
variables and the reasons behind any such correlations. In this case, the interviewer is the instrument for qualitative 
data collection. The criteria for selecting individuals to interview necessitates that they must have had an experience 
with online learning environments. This information could be obtained from students’ responses on the survey 
question where they report the number of online courses they have had. Because this approach is based on SDT of 
motivation and situated learning, selecting individuals who have diverse gaming experience should be considered to 
provide differing points of view. Graduate and undergraduate students who are at different levels of video game 
experience could hold different perspectives toward the pleasurability of gamification in online learning 
environments. To get a balanced view, the researcher can select some students whose answers on the attitudinal 
questions would represent more positive attitudes toward gamification and others who would be identified as 
exhibiting more negative attitudes toward gamification. This purposeful sampling of selecting outliers’ cases allows 
the researcher to target individuals who might be likely to provide in-depth information to further explain the 
quantitative research findings. 

The semistructured interview protocol works best for this approach because it can be seen as more of a 
conversation that draws out rich, in-depth data (Qu & Dumay, 2011). The data gathered through interviews must be 
understood within its social context, rather than considering the interview itself as a tool for collecting data (Qu & 
Dumay, 2011). This comprehensive understanding of the social context will help instructional designers and 
instructors to include applications of educational gamification in online courses. In this regard, the design of online 
courses can tackle various motivational issues of teaching and learning in online environments. Since interviews are 
best done when the interviewer approaches the interviewee with “a respect for and curiosity about what people say, 
a willingness to acknowledge what is not understood, and the ability to ask about what is not yet known” (Rubin & 
Rubin, 2005, p. 13), a semistructured approach will allow the interviewer to follow the course of conversation as it 
emerges through the dyadic exchange. 

A semistructured interview could start by asking some questions of the interviewee to initiate the discourse, 
such as: (a) “Tell me about your personal experience with video games.” (b) “Why did you perceive (item X) as the 
most pleasurable learning experience regarding instructions?” or (c) “Why did you perceive (item Y) as the least 
pleasurable learning experience regarding instructions?” However, the interviewer should let the participants lead 
the way with their own unique insight as well. For this reason, the interviewer must listen for cues from the 
interviewee to find areas where he/she can expand on the findings. 

The interviewer could use both scheduled and unscheduled probe techniques during the semistructured 
interviews to elicit as much information from the participant as possible (Qu & Dumay, 2011). Scheduled probes 
can help to elaborate on interesting or unusual comments made by the interviewee by asking follow-up questions 
like: “Can you tell me more about that?” and “Can you give an example?” A semistructured interview is not a 
neutral process, but “‘a situated event in which the interviewer creates the reality of the interview situation” (Qu & 
Dumay, 2011, p. 247). Due to the subjectivity that might be imposed by the interviewer, the interviewer should 
report the interview process and findings with thick descriptions. This part of the process must include in-depth 
details of the interview setting, context, participants, and the interviewer’s actions to facilitate the transferability of 
the findings into other contexts (Yilmaz, 2013). 


Conclusion 


Educational gamification is intended to design engaging learning experiences that can spur students’ 
motivation to learn. This type of persuasive design adapts the strategies that make video games pleasurable and fun 
into the learning task so students feel fully immersed in their learning. To accomplish this goal, educators and 
instructional designers in higher education institutions should explore what would make learning pleasurable and 
fun from students’ perspectives as a first step to designing effective educational gamification. Unfortunately, there is 
limited research that investigates students’ attitudes toward educational gamification before its implementation in 
online learning environments. Moreover, most of the studies that intended to analyze students’ needs lacked the use 
of validated tools that are based on theoretical foundation. This proposed approach is a research methodology to 


investigate graduate and undergraduate students’ attitudes toward gamification in online learning environments. This 
explanatory mixed-method approach employs both quantitative and qualitative methods to explore students’ 
perspectives toward the pleasurability of different learning experiences. 

The current approach is based on: (a) the concept of pleasurability, (b) the self-determination theory of 
motivation (SDT), and (c) situated learning. The concept of pleasurability refers to joy that students would 
experience while engaging in gamified online courses. This experience can be classified into nine categories: (a) 
control, (b) submission, (c) humor, (d) sensation, (e) visualization, (f) challenge, (g) discovery, (h) fellowship, (i) 
interaction. These nine categories are associated with three psychological needs that should be satisfied to make 
learning internally motivating activity to the students. According to SDT, these needs are autonomy, competence, 
and relatedness. Because learning is situated in an authentic context that encompasses knowledge and skills to be 
acquired, gamified online courses should provide opportunities for students to try multiple identities and be involved 
in rapid cycles of until reaching level of mastery. In this approach, the use of storytelling as a game element can 
support situating learning by giving students clear rules, presenting the course content as a story that engages and 
empowers students, and providing scaffolding to students as needed. 

This proposed approach employs two methods for investigation: (1) the Pleasurable Learning Experiences 
Scale (PLLEXs) (A, 2016), and (2) semistructured interviews. The PLLEXs is a survey that consists of four major 
parts: demographics, online learning experience, gaming experience, and attitudes toward gamification. The 
attitudinal questions are targeting students’ preferences for: (a) Instructions, (b) Instructors, (c) Activities, and (d) 
Learning Effectiveness. These four constructs constitute the overall students’ attitudes toward the pleasurability of 
educational gamification. According to this proposed approach, data collection and analysis are done in two phases 
sequentially. The first phase focuses on collecting quantitative data using PLLEXs to explore students’ attitudes 
toward video games and gamification quantitatively. The second phase focuses on collecting qualitative data by 
conducting semistructured interviews with selected students to examine their attitudes toward gamification 
qualitatively. The analysis of each phase would be carried out independently and the quantitative phase should 
inform the plan and the implementation of the qualitative phase. The collected data is integrated during the 
interpretation stage to provide in-depth information to the researcher. In this way, the approach could help to 
identify the key features of educational gamification from students’ perspectives, which would change the way 
students’ needs in online gamified education are identified and also change the way educational gamification could 
be implemented in online learning environments. 
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Abstract 


Twenty-five students enrolled in the asynchronous, online community college course Introduction to 
Education participated in this study. The course was taught at a community college in the Southeastern region of the 
United States. Students were placed in either a traditional discussion group or a reciprocal discussion group using a 
convenience sample based on enrollment in a section of the 16-week course. The reciprocal group served as the 
treatment group and received embedded reciprocal teaching training and used the four reciprocal teaching strategies 
and peer teaching in asynchronous discussion forums. The traditional group served as the control group and 
followed typical discussion forum protocol by posting in a teacher-led discussion forum and responding to two other 
students. Results showed the reciprocal discussion group had significantly higher-level thinking and deeper 
processing of course texts during discussions. The study found that reciprocal teaching could be successfully 
implemented in an online, asynchronous course. 


Keywords: reciprocal teaching, interaction, cognitive strategies, metacognitive strategies, peer teaching, 
asynchronous online course, and discussion forum. 


Introduction 


Reciprocal teaching is an interactive instructional procedure that improves students’ text comprehension 
skills through scaffolded instruction of four comprehension-fostering and comprehension-monitoring strategies 
(Palincsar & Brown, 1984; Rosenshine & Meister, 1994). The four reciprocal teaching strategies are predicting, 
questioning, clarifying, and summarizing (Palincsar & Brown, 1984, 1986; Palincsar, Brown, & Martin, 1987). 
Reciprocal teaching involves student-led instruction, modeling, practice, and feedback in metacognitive, self- 
monitoring, and evaluating strategies (Brown, Campione, & Day, 1981). According to the literature, reciprocal 
teaching leads to improved reading comprehension. It has been studied in many contexts with multiple subjects. 

In this study, reciprocal teaching was translated into an online, asynchronous course. Purposeful, strategy- 
rich discussion forums resulted and enabled students to negotiate meaning and deeply understand course texts. 
Reciprocal teaching supported the negotiation of meaning in a social learning atmosphere. Peer teaching resulted in 
generative processing through the reworking of a topic from the textbook into a lesson and questions for peers 
(Collins et al., 1989; King, 1991; Pressley et al., 1992; Rosenshine et al., 1996; Wood et al., 1990). The strategies 
and peer teaching engaged students in high-level, content-based discussions leading to deeper understanding course 
texts in the online, asynchronous context. 


11 


Following are the research questions that guided this study: 


1. To what extent did the type of discussion forum strategies, traditional or reciprocal, effect levels of thinking 
during posts in asynchronous discussion forums? 

2. To what extent did the type of discussion forum strategies, traditional or reciprocal, facilitate deeper 
understanding of the course textbook? 

3. To what extent were reciprocal teaching strategies and peer teaching implemented in online, asynchronous 
discussion forums? 

4. What impact did traditional discussion forums have on student reflections of the relationship between 
discussion forums and learning? 

5. What impact did reciprocal teaching have on student reflections of the relationship between strategies, peer 
teaching and learning? 


Methods and Data Sources 


Quasi-experimental, multiple methods were employed to compare the effects of traditional discussions and 
reciprocal teaching discussions. Outcome variables were level of thinking, understanding of course texts, online 
reciprocal teaching implementation, and students’ reflections on the relationship between discussions, strategies, and 
learning. The instruments used to measure the dependent variables included: the SOLO Taxonomy, Midterm and 
Final Exam, Reciprocal Teaching Rubric, and the Learning Reflection Tool. 

The SOLO taxonomy was first developed in 1989 to measure the quality of learning outcomes (Biggs, 
1989). The SOLO taxonomy was used to measure level of thinking in this study. Table | shows the levels of 
thinking and numeric score associated with each level of the SOLO taxonomy. 


Table 1. Modes and Levels of the SOLO Taxonomy 
Mode Structural level SOLO 


Next Level 5: Extended abstract The learner now generalizes the 
structure to take in new and more 
abstract features, representing a 
higher mode of operation. 


Target Level 4: Relational The learner now integrates the parts 
with each other, so that the whole 
has a coherent structure and 
meaning. 

Target 

Level 3: Multi-structural The learner picks up more and more 
relevant or correct features, but 
does not integrate them. 

Target 

Level 2: Uni-structural The learner focuses on the relevant 
domain, and picks up one aspect to 
work with. 


Previous Level 1: Pre-structural The task is engaged, but the learner 
is distracted or misled by an 
irrelevant aspect belonging to a 
previous stage or mode. 
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Results 


Results showed that reciprocal teaching strategies promoted significantly higher levels of thinking and 
deeper processing of course texts compared to traditional methods. Level of thinking SOLO scores for all discussion 
posts were higher in the reciprocal group (M = 4.1, SD = 0.4) than in the traditional discussion group (M = 2.8, SD = 
0.3), and the difference between these discussion groups was statistically significant, F(1,23) = 94.699, p < .001. 
There was a large effect size, 4? =. 81. Deeper understanding of course texts was measured by comparing scores of 
each group on a midterm and final exam. Total midterm scores were higher for the reciprocal group (M= 112.4, SD 
= 4.7) than for the traditional group (VM = 103.3, SD = 8.4). The differences between the groups was statistically 
significant F(1,22) = 9.619, p =.005. There was a large effect size, 7? = .30. Total final exam scores were higher for 
the reciprocal group (M@= 111.9, SD = 4.5) than for the traditional group (M = 98.7, SD = 12.9). The differences 
between groups on the total final exam was statistically significant, Welch’s F(1, 19) = 18.576, p< .01. 

Results also showed that the reciprocal teaching strategies and peer teaching could be implemented in 
discussion forums. To determine the effectiveness of strategy use in the reciprocal environment, students were rated 
on each strategy forum post using the reciprocal teaching rubric. Rubric grades were awarded 25 points for an 
exemplary post, 20 points for a proficient post, 15 points for a developing post and 10 points for a beginning post. A 
frequency distribution of all strategy posts combined showed that the mean of the 381 posts over the 11 weeks was 
24, (N = 381, M= 24, SD = 1.9). Only one student scored at the beginning level, four students scored at the 
developing level, 65 students scored at the proficient level, and 311 students scored an exemplary on the rubric in 
the overall strategy forum analysis. The peer-teaching forum was found to be a very effective way for students to 
lead class discussions. The peer teaching forum was conducive to carefully planned lesson content and discussion 
questions posted by students and provided time for students to formulate a well-developed lesson and questions. 

The qualitative analysis of the study was conducted through surveys where students reflected on the 
relationship between discussion forums, strategies and learning. The survey for the reciprocal group was divided by 
strategy to help gain insight into student perspectives on each strategy. Students felt predicting helped prepare them 
for reading. Students found questioning helpful, especially when the instructor or other students answered the 
questions they posted. Similarly, the responses in the survey related to clarifying showed that students found value 
in having a space to post items for which they needed clarification where they could receive answers from the 
instructor. Summarizing helped them understand, remember and organize thoughts. Students felt peer teaching was 
innovative, unique and fun. The enjoyed the challenge and looked forward to being the peer teacher. Overall both 
groups of students highly rated their discussion forum types. 


Discussion 


Adding to findings reported in previous literature, reciprocal teaching supports text comprehension. 
Reciprocal teaching in the context of an online, asynchronous community college course supported interactions that 
led to higher levels of thinking and deeper understanding of course texts. All three types of interaction were present 
in the reciprocal teaching implementation. Student-content, student-teacher, and student-student interaction 
occurred as students used cognitive and metacognitive strategies to process the content in the course textbook during 
discussions. Student-content interaction was present in all of the forums as student posts were based on the content 
in the course textbook. The high prevalence of student-content interaction ensured there was not a lack of initial 
understanding of content prior to engaging in online discussions. The peer teaching forum facilitated student-student 
interaction as students asked and answered questions and responded to each other. 

Relating back to the literature review, the results from this study showed that the full reciprocal teaching 
method could be effectively implemented in an online course leading to significantly higher levels of thinking and 
deeper processing of course texts. 
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Limitations 


The first limitation was the small sample size (7 = 25). There was a small number of students enrolled in 
the two sections of the course resulting in a small sample size. Research conducted with a larger sample size would 
increase the validity of the results and generalizability to all learners. A second limitation was the quasi- 
experimental design and sampling procedure. The study was conducted over two different semesters to obtain 
enough participants for a control and treatment group. Since the study was conducted over two semesters, there may 
have been a threat to external validity. A third limitation was the difficulty level of the course and textbook used in 
the study. Community colleges are two-year institutions and courses are not reflective of the difficulty level of 
bachelors or masters level courses. The course, Foundations of Education is a survey course and is not as difficult as 
more advanced upper-division courses. The introductory nature of the course may not have necessitated higher-level 
thinking. Another limitation was that there was not a pre-test or assessment of prior content knowledge given to 
participants. There was no way to determine whether the significant results of the reciprocal group were due to prior 
knowledge or higher aptitude than the traditional group. Another limitation came with the technology used by some 
of the peer teachers. Peer teachers were not trained to use the technology in Moodle and there was occasionally a 
struggle to get the lesson posted and viewed. Some of the students had difficulty accessing the lessons without 
intervention from the teacher. Finally, even though every effort was made for the ratings of discussion posts to be 
blind, the researcher and instructor could easily distinguish between the groups based on the questions asked in the 
forums. The familiarity with the course content and the instructor-led questions made it possible for the researcher 
and instructor to know the group from which the posts were written, possibly leading to experimenter bias. 


Conclusions 


This study was unique in that the full reciprocal teaching method was implemented in an online, 
asynchronous course with fidelity including the four strategies and peer teaching. The discussion forums provided a 
social setting so that students could share the responsibility of making meaning through social negotiation using 
generative learning strategies (Glaser, 1990; Vygotsky, 1978; Wittrock, 1974, 1990). The reciprocal teaching 
strategies and peer teaching fostered high levels of student interaction with the content and with the teacher and 
other students having a significant positive influence on learning. Peer teaching resulted in generative processing 
through the reworking of a topic from the textbook into a lesson and questions for peers (Collins et al., 1989; King, 
1991; Pressley et al., 1992; Rosenshine et al., 1996; Wood et al., 1990). 

Based on the promising findings of this study, practitioners may want to adopt reciprocal teaching in their 
online courses. The first step is to teach students how to participate in reciprocal teaching. Direct instruction on the 
strategies and peer teaching can be embedded in learning management system during the first weeks of class using 
videos, presentations, and documents. The instructor may model the strategies and act as the peer teaching the first 
week to show students what is expected and how to participate. Grading criteria and rubrics should be presented 
early in the class. 

There is a wide-open door for more studies in distance education focused on improving text comprehension 
through research validated strategies such reciprocal teaching. Researchers, instructional designers and teachers 
need to ensure that online courses are engaging, interactive and most of all optimally designed for learning. While 
this study was a step towards understanding how to increase learning through specific interaction strategies in an 
online, asynchronous course, there is a lot left to discover about learning in the realm of distance education. 
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Abstract 


Online learning continues to grow at unprecedented rates. In higher education over 7.1 million students 
took a course in 2013, which represented over 33.5% of the total student population. K-12 is growing at similar rates 
with over 275,000 students enrolled in full time virtual schools in 2011-2012 and over 2 million participating in at 
least one online course. Corporations are experiencing growth as well with technology based learning at 37.30% and 
online learning at 24.77%. 

Despite this unprecedented growth, there is little research on model formation and model validation for 
online learning. Two research-based models were created: SUCCESS and R2D2/C3PO to help project managers, 
instructional designers, and developers design and implement online learning solutions. Both of these models were 
created based on research findings using qualitative case study and design and development case study guidelines. 
Both models have a strong focus on culture, learning, and technology, and the change management strategies 
necessary for successful design and implementation of online learning in different contexts such as, K-12, higher 
education, healthcare, military, and corporate organizations. 

The purpose of this descriptive qualitative study was to describe the process of model creation for both 
models and to systematically validate the two models using a survey and The Nominal Group Technique (NGT) 
with expert instructional designers. Participants were instructional designers who are experts in online learning 
across industries. Five participants were asked to review materials on the models and provide comments on the 
models pre-meeting. Following the individual model review, three participants attended a 1 1/2-hour focus group 
discussion responding to and commenting on each element in the models. The focus group session was 
professionally facilitated, digitally recorded, and transcribed verbatim. 

The NGT is a five-step process whereby participants meet one another, generate ideas based on their pre- 
work review, share ideas with the group, discuss ideas, and vote and rank items based on original questions about 
the models. In addition to the discussion session, participants were asked to complete a survey ranking elements of 
the model using a Likert-scale. 

Data analysis included the use of descriptive statistics generated in EXCEL from the survey. Themes 
generated from the focus group discussion session were analyzed using NVIVO software, which was used as a 
centralized database for all study materials. A preliminary list of start codes based on the elements of the model was 
used to begin the data analysis and was supplemented with new emerging themes from the analysis. Similar analysis 
was done with the individual response sheets the participants submitted as part of the pre-work. Finally, the data 
analysis showed the results of the voting and ranking at the NGT focus group discussion session. Results of the 
analysis are presented as a narrative, as a matrix summary to show the level of support for each of the model 
components, and as potential updates to the models. 

The U.S. is behind other developed countries in many areas of academic education and students are finding 
it difficult to find gainful employment upon graduating from U.S. schools. The cost of education continues to 
escalate in the U.S. Online learning is a cost-effective way to deliver learning at affordable price points and often at 
faster rates than traditional classroom. Online education can be available 24X7X365 anywhere in the world where 
there is an internet connection. Companies and educational institutions are providing classes to diverse cultural 
populations around the world. Providing project managers, instructional designers, systems administrators, and 
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developers with research-based models that have been evaluated by experts will give these leaders the tools they 
need to successfully deliver online learning to cross cultural groups. 


Introduction 
Problem Statement 


Online learning continues to grow at unprecedented rates. In higher education over 7.1 million students 
took a course in 2013, which represented over 33.5% of the total student population (Allen & Seaman, 2014). K-12 
is growing at similar rates with over 275,000 students enrolled in full time virtual schools in 2011-2012 and over 2 
million participating in at least one online course (Allen & Seaman, 2014). Corporations are experiencing growth as 
well with technology based learning at 37.30% and online learning at 24.77% (Miller, 2012). 

Despite this unprecedented growth, there is little research on model formation and model validation for 
online learning (Richey & Klein, 2007; Richey, Klein, & Tracey, 2011). Two research-based models were created: 
eSUCCESS (Armstrong, 2007; 2008; 2016b) and R2D2/C3PO (Armstrong, 2014; 2016a) to help project managers, 
instructional designers, and developers design and implement online learning solutions. Both of these models were 
created based on research findings using qualitative case study and design and development case study guidelines. 
Both models have a strong focus on culture, learning, and technology, and the change management strategies 
necessary for successful design and implementation of online learning in different contexts such as, K-12, higher 
education, healthcare, military, and corporate organizations. The specific problem that was the focus of this study is 
validation of the two research based models. 


Purpose Statement 


The purpose of this descriptive qualitative study was to describe the process of model creation for both 
models and to systematically validate the two models using a survey and The Nominal Group Technique (NGT) 
with expert instructional designers. For the purpose of this study, the NGT was based on Potter, Gordon and 
Hamer’s (2004) adaptation of the technique. Participants were instructional designers and instructional 
technologists, who are experts in online learning across industries. Inclusion criteria included more than 5 years of 
instructional design/instructional technology experience and a doctoral degree in Instructional Design, Instructional 
Technology, or related field. 


Research Questions 


The research questions were focused on validating the two models based on the experiences of expert 
instructional designers and align with Yin’s (2014) assertions that case study questions should be “how” and “why” 
questions and Thomas’ (1983) expert requirements that the questions be clear and stimulating for the participants. 


Research Question 1. How do instructional designers believe the eSUCCESS model serves as a valid 
framework for design and implementation of online learning programs? Why do they hold these beliefs? 

Research Question 2. How would the instructional designers augment the model? 

Research Question 3. How do instructional designers believe the R2D2/C3PO model serves as a valid 
framework for design and implementation of online synchronous learning programs? Why do they hold these 
beliefs? 

Research Question 4. How would the instructional designers augment the model? 


Methodology and Design 


The methodology was qualitative and the design descriptive case study. Data collection techniques 
included document review of the models, expert survey responses, and a focus group interview using the NGT 
technique. Six participants were asked to review materials on the models and provide comments and complete a 
survey on the models pre-meeting. Five agreed to participate. Following the individual model review, three of the 
five participants attended a 1 % hour NGT focus group discussion responding to and commenting on each element in 
the models. The focus group session was professionally facilitated, digitally recorded, and transcribed verbatim. 

Jones and Hunter (1995) emphasized the importance of recruiting participants who are subject matter 
experts on the topic. Experienced instructional designers having at least 5 years instructional design experience and 
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an advanced degree in instructional design, instructional technology, educational technology, or an equivalent 
degree were recruited. A purposive, convenience sample (Miles & Huberman, 1954) was used based on 25+ years of 
networking experience with instructional designs who met the inclusion requirement. 

Several instruments were employed during the data collection process: 1) pre-data collection questionnaire, 
2) pre-focus group model evaluation and survey, and a 3) focus group protocol based on the NGT. The model 
evaluation, model survey, and focus group data were used for convergence of data following Yin’s (2014) call for 
triangulation to ensure the validity of the data. The Model Evaluation Surveys required participants to write down 
their own ideas that come to mind as they are reviewed each of the models. The Model Evaluation Survey also 
required the participants to rank the elements of the models using a Likert Scale. 

The NGT is a five-step process whereby participants meet one another, generate ideas based on their pre- 
work review, share ideas with the group, discuss ideas, and vote and rank items based on original questions about 
the models. Delbecq and van de Ven (971) stressed the importance of using an expert in the topic area during the 
NGT session and Gallagher et al. (1993) stressed that the facilitators must be highly skilled in group facilitation 
techniques. Both facilitators were experts on the topic having worked on the design and development of the models 
and both are trained and endorsed Dale Carnegie trainer/facilitators. The focus group was held online using a virtual 
classroom technology, GoToMeeting. Prior to the NGT focus group and following receipt of Informed Consent and 
Northcentral University IRB approval, participants received an introduction to the NGT protocol and several 
published articles that explained the models. Participants were then asked to rate each item in the models on the 
Model Evaluation Survey using a Likert scale from 1-4 to rank the importance of each element in the model, with 1 
as unimportant, 2 of little importance, 3 important, and 4 very important. Space was available for comments and 
suggestions. Estimated time was 1 2 hours. Following receipt of the surveys, an NCT focus group session was 
scheduled. Each of the steps in the NCT protocol was followed: 

Step 1. Introduction and explanation. A focus group session was held online using the GoToMeeting 
virtual classroom technology. The session began with an introduction and explanation of the purpose, a review of 
the agenda (appendix C) and an overview of the procedures for the meeting (5 minutes); 

Step 2. Silent generation of ideas. The second step for each model was quiet time for silent generation of 
ideas. Participants were asked to reflect on each model separately and generate ideas for what components that 
would add to each element of the model and to the model itself and to write these ideas on a sheet of paper (10 
minutes); 

Step 3. Sharing ideas. Participants were asked to share their ideas based on their review of the materials in 
Step 2 and prior to the NCT session. Brookfield and Preskill’s (1999) Circle of Voices protocol (appendix D) was 
used so that each person could further explain their ideas, initially without interruption. In this step, no debate was 
allowed and everyone had an opportunity to make an equal contribution to the discussion (3-5 minutes per 
participant). Total time 15 minutes. 

Step 4. Group discussion. Continuing with the Circle of Voices protocol, after each participant had shared 
their individual ideas, group discussion was opened up and cross talk allowed. (15 minutes) 

Step 5. Voting and Ranking. Facilitators reviewed suggestions for enhancement to each model and 
presented the suggestions to the group. Participants voted on each suggestion and the results are in the results and 
findings. Each participant later emailed their silent generation of ideas sheets back to the facilitators. 


Data analysis included the use of descriptive statistics generated in EXCEL from the survey. Results were 
calculated for the average, mean, mode, and standard deviation of each element of the models. Themes were 
generated from the following: 1) comment section of the surveys, 2) NCT focus group discussion session, and 3) 
silent generation of ideas sheets. Each data collection vehicle was analyzed using NVIVO software, which was used 
as a centralized database for all study materials. A preliminary list of start codes based on the elements of the model 
was used to create parent and child nodes and start the data analysis. The preliminary list of start codes, transferred 
to parent and child nodes, was supplemented with new emerging themes from the analysis. Finally, in the data 
analysis are the results of the voting and ranking at the NGT discussion session. Results of the analysis are presented 
as a narrative and as a matrix summary to show the level of support for each of the model components. This data 
will be used to update the models. 


Models Validated 
The goal of this study was to use expert subject matter experts to validate two research-based models. The 


first model was the R2D2/C3PO model based on Bonk and Zhang’s (2006) R2D2 model that focused on 
constructivist strategies for developing online instruction. Armstrong (2014; 2016a) expanded this model during 
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design and development case study research using a blend of synchronous and asynchronous delivery in higher 
education classes (Armstrong & Thornton, 2012). Five components were added to Bonk and Zhang’s original model 
along with tools, techniques, strategies, and activities for each component of the model. 


R2D2/C3PO. 


Table 1 shows the R2D2/C3PO model as an expansion of the original Bonk and Zhang (2006) R2D2 
model. Each component is described, followed by instructional strategies and activities that can be used for each 
component. Finally, synchronous tools to support each of the learning activities and instructional strategies are 


described. 


Table 1. R2D2/C3PO Components, Instructional strategies/learning activities, synchronous tools 


Reflect/Writing/Sharing 


Creating participant manuals for each 
Live synchronous event. 


Listening to audio materials online or 
offline. 


Synchronous Expert 


Lectures/Presentations/Tutorials 
Demonstrations. 


Time allocated for reflection: 
1. Pre-work 
2. During synchronous event. 
3. Between synchronous events 

(multi-event course) 

Small Group Discussion. 

Focus Group Discussion. 

Online Role Play. 

Self-Assessment. 


Online Quizzes. 


Practice. 


R2D2/C3PO Component Instructional Strategies/Learning Synchronous Web-Conferencing Tools 
Activities 
Read/Listening Reading materials online or offline. Downloadable participant manuals as pdf 


and/or word files. 
Downloadable audio files. 
Downloadable video/multimedia files. 


“Live” synchronous facilitator/faculty 
presentation/demonstrations/lectures. 


Facilitator/faculty tools include 
whiteboards, chats, application sharing, file 
transfer, share pods, screen sharing, 
PowerPoint slides, polls, and note boards for 
announcements, Q&A, and FAQs. 


Links to websites. 


Downloadable participant manuals used to 
write reflections and journal. 


Chat pods used for sharing reflections with 
other participants. 


Polls used to capture self-assessment and 
compare anonymously with other 
participants. 


Breakout rooms used for small group 
discussions, debates, mock trials, role play, 
practice with partners, and focus groups. 


Display 


Visual Representations including: 
pictures, diagrams, graphs, 


multimedia, video, charts, animations. 


Demonstrations. 
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Downloadable participant manuals with 
visual representations of key concepts. 


Facilitator/faculty “live” presentation using 
a share pod with a PowerPoint presentation 
with robust visuals to help explain concepts. 


Visual of the facilitator/faculty presenting or 
demonstrating “live,” which enhances 
teacher presence. 


Links to videos on YouTube, TED Talks, 
Khan Academy etc. 


Collaboration, Cooperation, and Trust. 


Doing Caselettes (short cases). Downloadable participant manuals with 
case, project, collaboration, and 
Case Studies. apprenticeship instructions. 
Practice. Live facilitator/faculty presentation, 
demonstration, and modeling for all 
Project-based learning. activities. Time allocated for reflection and 
Q&A. 
Collaborative group project, paper, 
etc. Breakout rooms with whiteboards and chat 
pods for individual and collaborative work. 
Cognitive Apprenticeship. Polls available, when needed. 
Assessment. Breakout rooms for practice with partners or 
in larger groups, such as triads. 
“Live” individual or group presentations in 
the main room. 
Coaching Cognitive Apprenticeships. Facilitator/faculty feedback through “live” 
voice, chat, and private chat. 
Scaffolding. 
Peer coaching in breakouts and private 
Interactive feedback. chats. 
Conviviality Ground Rules to encourage Participatory establishment of ground rules 


using “live” discussion, chats, whiteboard 
tools, and polls. 


Critical Incident 
Questionnaire (CIT) 


Use CIT for immediate student 
feedback between instructional events. 


Downloadable form or Anonymous Poll. 


Planning/Organization 


Fast Start 

Quick Reference 
Participant Manual 
Timeline for all Activities. 


Time for Reflection during event and 
between events. 


Fast Start and Quick Reference Guides on 
how to use the web conferencing tools. 


Participant manual containing content and 
exercise during the live session. 


Facilitator/faculty manual with suggested 
timelines for all activities. 


Reflection time built into all events. 


Reprinted with permission from Nova Science Publishers, Inc from: Constructivist Instructional Strategies for 
Synchronous Web Conferencing: Synchronous Constructivist Instructional Strategies for the 21st Century. In Dale 
H. Eberwein Editor (Ed.). Advocacy for Change in Educational Culture (Chapter 2) (2016), Author: Ann 
Armstrong. ISBN: 978-1-63484-206. 


eSUCCESS. 


Following Christensen (2006) stages of model building Armstrong (2007; 2008; 2016b) observed Chief 
Learning Officers (CLOs) of some of the largest corporations in the world and interviewed these CLOs as they 
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rolled out large-scale learning management systems and online web-based learning courses for several hundred 
thousand employees. Following observation and interviews, data was categorized into themes and associations and 
connections reported (Armstrong, 2007; Armstrong, 2008). The analysis revealed five major categories and 30 
indicators that ultimately became eSUCCESS. Reducing the data further, eight tenets were identified that embodied 
how CLOs described the necessary elements for successful implementation of an eLearning system. Table 2 shows 
the tenets and their descriptions. 


Table 2. SUCCESS Tenets with Descriptions 


Tenet | Tenet Name Description 


# 


1 eXecutive Sponsorship Time 

Money 

Resources 

Communications 

Vision 

Values 

Change Management Strategy 


2 Support from the Organization Line of Business Executive 

Learning Executive 

IT Executive 

Direct Manager Involvement 

Project Manager for eLearning Initiatives 
Strong Technology Infrastructure 

Help Desk 

“Built into the Organization” 
Communities of Practice 

Social Networks 


3 Understand and Motivate the Learner Benefit Communicated 

Included in Performance Plan 
Granted Additional Resources 
Continuing Education Requirement 
“Peer Pressure” 

“Mandated” by the Regulators 
Pre-requisite to Live Training Event 
Threat of Job Loss 

Certification and Accreditations 


4 Culture Fosters Learning Systems for Continuous Learning and 
Sharing 

Promote Dialogue and Inquiry 
Collective Mission 

Leader 

Collaboration and Team Learning 
Connecting to the Environment 


5 Content is Relevant to the Learner and Job and Task Specific 
Organization CEUs, PDU, CPEs 
Certifications 


Branded to the Culture and Environment 
Recognized SMEs/Experts 
Facilitating/Teaching 

“Hot Topics” in the Industry 
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6 Evaluate and Assess Kirkpatrick Level | 


Kirkpatrick Level 2 
Kirkpatrick Level 3 
Kirkpatrick Level 4 
7 Structure of Program is Engaging, Interactive, On-the-Job Training (OJT) 
and Blended Coaching and Mentoring 


Web-based Technology 
Traditional Classroom 


8 Simulate the Work Environment and Work Goal-based Scenarios 
Tasks Problem-based Learning 
Case Studies 
Caselettes 


Reprinted with permission from Nova Science Publishers, Inc from: Towards a Theory of Enablers to eLearning 
Adaptation: eSUCCESS, a research Based Framework. In Dale H. Eberwein Editor (Ed.). Advocacy for Change in 
Educational Culture (Chapter 3) (2016), Author: Ann Armstrong. ISBN: 978-1-63484-206. 


Results and Findings 


To meet the NGT requirement that evaluators were experts in the field, a pre-data collection questionnaire 
was used to ensure each participant met the study inclusion criteria. Inclusion criteria included more than 5 years of 
instructional design/instructional technology experience and a doctoral degree in Instructional Design, Instructional 
Technology, or related field. Table 3.0 shows participant demographics. All experts had at least 15 years 
instructional design experience with three having over 20 years. All experts had doctoral degrees in instructional 
design, instructional technology, or a related field. All experts worked in field where they applied instructional 
design and instructional technologies. Table 3.0 details participant demographics across the inclusion criteria for the 
study. 


Table 3.0. Participant Demographics 


Participant Years | Title Degree | Concentration Conferences 
# in 
Field 
Pl >20 Instructional Director EdD Leadership in Education USDLA, 
for Training and FGDLA 
Education 
P >20 Instructional Designer EdD Human Resource Developmey Sloan-C 
Distance 
Education 
P3 15-20 | Design Consultant PhD Applied Technology Training) ATD, ISPI, 
Development AECT 
P4 15-20 | Instructional Design PhD Educational Technology AECT, ATD 
Faculty OLC 
PS >20 Program Director, PhD Computing and IDC, PLN 
Instructional Technology in 
Technology Education 
Results from Participant Responses to R2D2/C3PO Model Evaluation Survey. Table 4.0 shows the 
results from participant evaluations of the R2D2/C3PO Model Evaluation Survey (Appendix A). Each participant 


was asked to rank each component from 1-4, with | as unimportant, 2 of little importance, 3 important, and 4 very 
important. All components of the model received at least a 3, important, with the exception of 1b (create participant 
manuals for live synchronous events), 2b (reflection time allocated during synchronous event), 2e (focus group 
discussions), and 2f (online role play). 
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Table 4.0. Rankings by Participants: R2D2/C3PO Model Evaluation Survey 


Ranking by Participants 1 2 3 4 5 Mean | Mode | SD 
1 Read/Listen 
la__| Read materials online or offline 4 3 3 4 4 3.6 4 44 
1b | Create participant manuals for “live” 2 2 2 2 4 2.4 2 69 
synchronous events. 
lc | Listening to audio materials online or | 3 3 3 3 4 3.2 3 BS 
offline. 
ld | Facilitator for lectures/presentations/ 3 3 3 3 4 3.2 3 35 
Tutorials/demonstrations is a subject 
matter expert 
2 Reflect/Writing/Sharing 
2a__| Time allocated for pre-work 3 4 3 3 4 3.4 3 .44 
2b | Reflection time allocated during Z 4 2 2 4 2.8 2 87 
synchronous event 
2c | Reflection time allocated between 3 4 3 3 4 3.4 3 44 
events (multi-day) 
2d __| Small group discussions 3 3 3 3 4 3.2 3 35 
2e | Focus group discussions 2 3 2 2 4 2.6 2 71 
2f | Online role play 2 3 2 2 4 2.6 2 71 
2g _| Self-assessment 3 3 4 4 4 3.8 4 35 
2h__| Online quizzes 3 3 4 3 4 3.4 3 44 
21 Practice 4 3 4 4 4 3.8 4 35 
3 Display 
3a __| Visual representations, such as: 4 3 3 4 4 3.6 4 44 
pictures, diagrams, graphs, multi- 
media, video, charts, animations 
3b__| Demonstrations 4 3 3 4 4 3.6 4 4 
4 Doing 
4a__| Caselettes (short cases) 3 3 2 3 4 3.0 3 53 
4b__| Case studies 3 2 2 3 4 2.8 3 .64 
4c__| Practice 4 3 4 4 4 3.8 4 35 
4d __| Project-based learning 4 3 4 4 4 3.8 4 Se) 
4e | Collaborative group project 4 2 2 4 4 3.2 4 .87 
4f | Cognitive apprenticeship 4 2 3 4 4 3.4 4 71 
4g__| Assessment 4 3 4 4 4 3.8 4 35 
5 Coaching 
5a__| Cognitive apprenticeships 4 2 3 4 4 3.4 4 71 
5b__| Scaffolding 4 3 3 4 4 3.6 4 44 
5c__| Interactive feedback 4 3 4 4 4 3.8 4 35 
6 Conviviality 
6a | Ground Rules to encourage 4 2 4 4 4 3.4 4 71 
collaboration, cooperation, and trust 
7 Critical Incident Technique (CIT) 
7a | Use CIT for feedback between multi- | 4 2 2 4 4 3.2 4 87 
day instructional events 
8 Planning/Organization 
8a__| Fast Start 4 2 3 4 4 3.4 4 71 
8b | Quick Reference 4 2 3 4 4 3.4 4 71 
8c | Participant Manual 4 3 3 4 4 3.6 4 44 
8d__| Timeline for all activities 4 2 3 4 4 3.6 4 .69 
8e | Time for Reflection during events and | 4 3 3 4 4 3.6 4 44 
between events 
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Results from participant responses to eSUCCESS Model Evaluation Survey. Table 5.0 shows the 


results from participant evaluations of the eSUCCESS Model Evaluation Survey (Appendix B). Each participant 
was asked to rank each component from 1-4, with 1 as unimportant, 2 of little importance, 3 important, and 4 very 
important. All components of the model received at least a 3, important, with the exception of 3h (threat of job loss) 
and 7d (traditional classroom). 


Table 5.0. Rankings by Participants: SUCCESS Model Evaluation Survey 


Question 1 2 3 4 5 Mean | Mode | SD 
1 eXecutive Sponsorship 
la__| Time 3 3 4 3 4 3.2 3 64 
lb | Money 3 3 4 3 4 3.4 3 44 
Ic | Resources 3 3 4 3 4 3.4 3 4 
1d | Communications 3 3 4 3 4 3.4 3 44 
le | Vision 4 4 4 4 4 4 4 0 
lf | Change Management Strategy 4 3 4 4 4 3.8 4 35 
2 Support from the Organization 
2a__| Line of Business Executive 3 4 4 3 4 3.6 4 .44 
2b__| Learning Executive 3 4 3 3 4 3.4 3 4 
2c__| IT Executive 3 4 2 3 4 3.3 3 .64 
2d __| Direct Management Involvement 3 4 3 3 4 3.4 3 4 
2e | Project Manager for eLearning 4 4 2 4 4 3.6 4 .69 
Initiatives 
2f | Strong Technology Infrastructure 4 4 4 4 4 4 4 0 
2g | Help Desk 4 4 4 4 4 4 4 0 
2h__| Built into the Organization 4 4 3 4 4 3.8 4 35 
21 Social Networks 2 4 2 3 4 3.2 3 .64 
3 Understand and Motivate the 
Learner 
3a__| Benefits Communicated 4 3 4 4 4 3.8 4 35 
3b__| Included in Performance Plan 4 2 4 4 4 3.8 4 35 
3c__| Granted Additional Resources 4 2 4 4 4 3.6 4 .69 
3d | Continuing Education Requirements 4 2 3 4 4 3.4 4 71 
3e | “Peer Pressure” 3 2 3 3 4 3.0 3 53 
3f_ | Mandated by the Regulators 3 3 4 3 4 3.4 3 44 
3g__| Pre-requisite to Live Training Event 4 4 3 4 4 3.8 4 35 
3h__| Threat of Job Loss 1 2 4 1 4 2.4 1 1.25 
3i Certification and Accreditations 3 3 3 3 4 3.2 3 35 
4 Culture Fosters Learning 
4a | Systems for Continuous Learning and | 3 4 3 3 4 3.4 3 44 
Sharing 
4b __| Promote Dialogue and Inquiry 3 3 3 3 4 3.2 3 35 
4c __| Collective Mission 3 3 3 3 4 3.2 3 35 
4d | Leader 3 4 4 3 4 3.6 4 44 
4e | Collaboration and Team Building 3 3 4 3 4 3.4 3 44 
4f _| Connecting to the Environment 3 3 4 3 4 3.4 3 44 
5 Content is Relevant to the Learner 
and Organizations 
5a__| Job and Task Specific 4 3 4 4 4 3.8 4 35 
5b__| CEUs, PDUs, CPEs 4 2 3 4 4 3.4 4 71 
5c __| Certifications 4 2 3 4 4 3.4 4 71 
5d__| Branded to the Culture and 3 2 3 3 4 3.0 3 53 
Environment 
Se | Recognized SMEs/Experts 4 2 2 4 4 3.0 4 87 


Facilitating/Teaching 
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5f__| “Hot Topics” in the Industry 2 2 4 4 4 3.2 4 .87 
6 Evaluate and Assess 
6a__| Kirkpatrick Level 1 2 2 2 2 4 2.4 2 .69 
6b__| Kirkpatrick Level 2 2 2 3 2 4 2.6 2 71 
6c Kirkpatrick Level 3 4 3 3 4 4 3.6 4 44 
6d | Kirkpatrick Level 4 4 3 4 4 4 3.8 4 35 
7 Structure of Program is Engaging, 

Interactive, and Blended 
7a | On-the-Job Training (OJT) 4 3 3 4 4 3.6 4 44 
7b | Coaching and Mentoring 4 3 3 4 4 3.6 4 44 
7c | Web-based Technology 4 2 3 4 4 3.4 4 71 
7d | Traditional Classroom 1 2 3 1 4 2.2 1 1.07 
8 Simulate the Work Environment 

and Work Tasks 
8a __| Goal-based Scenarios 4 3 4 4 4 3.8 4 35 
8b | Problem-based Learning 4 3 4 4 4 3.8 4 35 
8c__| Case Studies 4 3 3 4 4 3.6 4 44 
8d_ | “Caselettes” 4 3 2 4 4 3.4 4 71 


Results of the Focus Group 


Results from the NGT focus group yielded important addition suggestions for the two models which are 
presented in this section. The suggestions for the R2D2/C3PO follow and each is listed under the appropriate 
component in table 3 in bold. 

Read/Listening. Adding “live authors” to the synchronous expert lectures was suggested based on 
experience from several of the experts. 

Reflect/Writing/Sharing. Recommendations were made to enhance the reflection time by including “silent 
time” for reflection along with questions that encourage deep thought, such as “what was learned,” “how it was 
learned,” and “how the learning can be applied.” 

Display. Recommendations to enhance the display component included additional technologies such as 
virtual worlds and mobile technology, along with detailed demonstration techniques such as process steps, concept 
maps, logic steps, and using real-world work examples. There was a recommendation for providing instructor live 
video-based feedback of work products. 

Doing. Recommendation for adding adaptive learning and gaming to enhance interactivity and student 
engagement were suggested. 

Conviviality. Adding to the model a focus on support for the instructor/facilitator by focused attention on 
instructor/facilitator workload pre-and post-learning event, as well as during the learning event. 

Planning and organization. The importance of a component for both formative and summative 
assessment was emphasized, which can be readily accomplished through use of polls and observation. Adding a 
Facilitator/Instructor Manual that aligns with the Participant Manual would enhance the model and allow for greater 
consistency in delivery. 


Table 3. R2D2/C3PO Components Updated, Instructional strategies/learning activities, synchronous tools 


R2D2/C3PO Component Instructional Strategies/Learning Synchronous Web-Conferencing Tools 
Activities 
Read/Listening Reading materials online or offline. Downloadable participant manuals as pdf 
and/or word files. 

Creating participant manuals for each | Downloadable audio files. 

Live synchronous event. Downloadable video/multimedia files. 


Listening to audio materials online or | “Live” synchronous facilitator/faculty and 
offline. “Live Author” 
presentation/demonstrations/lectures. 
Synchronous Expert Lectures/” Live 
Authors”/Presentations/Tutorials 
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Demonstrations. 


Facilitator/faculty tools include 
whiteboards, chats, application sharing, file 
transfer, share pods, screen sharing, 
PowerPoint slides, polls, and note boards for 
announcements, Q&A, and FAQs. 


Links to websites. 


Reflect/Writing/Sharing 


Time allocated for reflection: 
1. Pre-work 
2. During synchronous event. 
3. Between synchronous events 
(multi-event course) 


“Silent Time” allowed for deep 
reflection 


In-depth Reflections 
What was learned? 
How learned? 
How learning can be 
applied? 
Small Group Discussion. 
Focus Group Discussion. 
Online Role Play. 


Self-Assessment. 


Online Quizzes. 


Downloadable participant manuals used to 
write reflections and journal. 


Chat pods used for sharing reflections with 
other participants. 


Polls used to capture self-assessment and 
compare anonymously with other 
participants. 


Breakout rooms used for small group 
discussions, debates, mock trials, role play, 
practice with partners, and focus groups. 


Practice. 

Display Visual Representations including: Downloadable participant manuals with 
pictures, diagrams, graphs, visual representations of key concepts. 
multimedia, video, charts, animations, 
virtual worlds, mobile technology. Facilitator/faculty “live” presentation using 

a share pod with a PowerPoint presentation 
Demonstrations. with robust visuals to help explain concepts. 
Process Steps 
Concept Maps Visual of the facilitator/faculty presenting or 
Logic Steps demonstrating “live,” which enhances 
Real-world Work teacher presence. 
Examples 
Links to videos on YouTube, TED Talks, 
Instructor live video-based feedback | Khan Academy etc. 
of work products and student work. 
Doing Adaptive learning. Downloadable participant manuals with 


Caselettes (short cases). 


Case Studies. 


case, project, collaboration, and 
apprenticeship instructions. 


Live facilitator/faculty presentation, 
demonstration, and modeling for all 
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Gaming. 
Practice. 
Project-based learning. 


Collaborative group project, paper, 
etc. 


Cognitive Apprenticeship. 


activities. Time allocated for reflection and 
Q&A. 


Breakout rooms with whiteboards and chat 
pods for individual and collaborative work. 


Polls available, when needed. 


Breakout rooms for practice with partners or 
in larger groups, such as triads. 


“Live” individual or group presentations in 


Collaboration, Cooperation, and Trust. 


Consideration for 
instructor/facilitator workload in 
planning event. 


Assessment. the main room. 
Coaching Cognitive Apprenticeships. Facilitator/faculty feedback through “live” 
voice, chat, and private chat. 
Scaffolding. 
Peer coaching in breakouts and private 
Interactive feedback. chats. 
Conviviality Ground Rules to encourage Participatory establishment of ground rules 


using “live” discussion, chats, whiteboard 
tools, and polls. 


Critical Incident 
Questionnaire (CIT) 


Use CIT for immediate student 


feedback between instructional events. 


Downloadable form or Anonymous Poll. 


Planning/Organization 


Fast Start. 

Quick Reference Guide. 
Participant Manual. 
Timeline for all Activities. 


Time for Reflection during event and 
between events. 


Evaluation and Assessment. 
Formative 


Summative 


Facilitator Manual. 


Fast Start and Quick Reference Guides on 
how to use the web conferencing tools. 


Participant manual containing content and 
exercise during the live session. 


Facilitator/faculty manual with suggested 
timelines for all activities. 


Reflection time built into all events. 


Self and group assessments in polls. 


Reprinted with permission from Nova Science Publishers, Inc from: Constructivist Instructional Strategies for 
Synchronous Web Conferencing: Synchronous Constructivist Instructional Strategies for the 21st Century. In Dale 
H. Eberwein Editor (Ed.). Advocacy for Change in Educational Culture (Chapter 2) (2016), Author: Ann 
Armstrong. ISBN: 978-1-63484-206. 


The suggestions for the eSUCCESS model follow and each is listed under the appropriate component in 


table 4 in bold. 


Tenet 1: eXecutive sponsorship. Adding mission to the description of vision and values was considered 


important. 
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Tenet 2: Support from the organization. More specificity around the concept of “built into the 
organization,” should be added including: needs assessment, continuous improvement in updated courses, and focus 
on learning transfer. 

Tenet 3: Understand and motivate the learner. Going beyond a performance plan to a 
development plan that focused on motivation training and remediation was recommended. 

Tenet 4: Culture fosters learning. Making sure that all leaders have leadership training and are 
required to attend various trainings with the “rank and file.” Emphasis on “futuring” and visioning and change 
management strategies should be included. Offering badges was recommended for fostering extrinsic motivation. 

Tenet 6: Evaluate and assess. Add Philips Level 5, Return on Investment (ROI), Return on 
Training (ROT), and continuous program evaluation and continuous needs assessments. 

Tenet 7: Structure of program is engaging, interactive, and blended. Add mobile 
technologies. 

Tenet 8: Simulate the work Environment and work tasks. Add guided practice with feedback, 
Ted Talks, YouTube videos, gaming, and virtual worlds. 


Table 4. SUCCESS Tenets Updated with Descriptions 
Tenet | Tenet Name Description 


# 


1 eXecutive Sponsorship Time 

Money 

Resources 

Communications 

Vision 

Mission 

Values 

Change Management Strategy 


2 Support from the Organization Line of Business Executive 
Learning Executive 
IT Executive 
Direct Manager Involvement 
Project Manager for eLearning Initiatives 
Strong Technology Infrastructure 
Help Desk 
“Built into the Organization” 
v Needs Assessment 
¥ Continuous Improvement-Course 
Updates 
Y Learning Transfer 
Communities of Practice 
Social Networks 


3 Understand and Motivate the Learner Benefit Communicated 
Included in Performance Plan 
Included in Development Plan 

¥ Motivation Training 

¥ Remediation 
Granted Additional Resources 
Continuing Education Requirement 
“Peer Pressure” 
“Mandated” by the Regulators 
Pre-requisite to Live Training Event 
Threat of Job Loss 
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Certification and Accreditations 
Badges 


Culture Fosters Learning 


Systems for Continuous Learning and Sharing 
Promote Dialogue and Inquiry 
Collective Mission 
Leader 

¥ Leadership training 
Collaboration and Team Learning 
Leaders/Managers attend training with 
rank and file 
Connecting to the Environment 
Futuring and Visioning 
Change Management Strategies 


Content is Relevant to the Learner and 
Organization 


Job and Task Specific 

CEUs, PDU, CPEs 

Certifications 

Branded to the Culture and Environment 
Recognized SMEs/Experts 
Facilitating/Teaching 

“Hot Topics” in the Industry 


Evaluate and Assess 


Structure of Program is Engaging, 
Interactive, and Blended 


Kirkpatrick Level 1 

Kirkpatrick Level 2 

Kirkpatrick Level 3 

Kirkpatrick Level 4 

Phillips Level 5-ROI 

Return on Training — ROT 
Continuous Program Evaluation 
Continuous Needs Assessment 
On-the-Job Training (OJT) 
Coaching and Mentoring 
Web-based Technology 
Traditional Classroom 

Mobile Technologies. 


Simulate the Work Environment and Work 
Tasks 


Goal-based Scenarios 
Problem-based Learning 

Case Studies 

Caselettes 

Guided practice with feedback 
Ted Talks 

YouTube Videos 

Gaming 

Virtual Worlds 


Significance of the Study 


Many models in instructional design and instructional technology and media are developed based on 


experience in practice and/or hearsay, rather than on rigorous empirical study (Richey, Klein, & Tracey, 2011). 
Richey and Klein (2007) explained that these models should be validated using design and development research. 
Both the R2D2/C3PO model and the eSUCCESS model emerged from rigorous research. The goal of this study was 
to further validate the models by intensive review by instructional designers and instructional technologists who had 
extensive and intensive experience in the field as well as doctoral degrees in the field. The models can serve as a 
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guide for practitioners as the seek to develop online learning programs. The expert validation strengthens the models 
and their use in practice. 


Conclusion 


Expert validation confirmed that the major components of the model were predominately important or very 
important, with a small fraction being of little importance. While the experts did not add major elements to either 
model, minor enhancements were made to most components of both the R2D2/C3PO and eSUCCESS models and 
will further augment the models and provide stronger frameworks for professionals to use as a guide for 
implementing online learning and synchronous online learning in their organizations. Two data collection 
techniques, expert Survey and expert NGT focus group, were primary in providing trustworthiness, transferability, 
credibility, confirmability, and dependability to the results and findings. 


Future Research 


The field of Instructional Design and Instructional Technology is relatively new and growing rapidly as 
technology is changing at an ever-increasing rate. Often models, frameworks, and processes are created based on 
experience in practice or limited research. Future research using a quantitative correlational analysis of the different 
components of both models would further validate the major categories and components of each model, thereby, 
enhancing their importance to researchers and practitioners alike. Applying the models in practice and reporting on 
the findings in a design and development model research study would be another way to extend the research. 
Additions to model components or categories from that research would serve to further enhance the instructional 
design knowledge base. 
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Appendix A 
MODEL EVALUATION SURVEY 1 


R2D2/C3PO Model 


Please rate each element in the R2D2/C3PO model as to its importance in the model on a scale of | to 4. 1 is unimportant. 
2 is of little importance. 3 is important. 4 is very important. 


Question 1 2 3 4 
1 Read/Listen 

la Read materials online or offline 

1b | Create participant manuals for “live” 
synchronous events. 

lc | Listening to audio materials online or 
offline. 

ld_ | Facilitator for lectures/presentations/ 
Tutorials/demonstrations is a subject 
matter expert 

le | Management usually keeps us 
informed about things we want to 
know 


2 Reflect/Writing/Sharing 
2a Time allocated for pre-work 


2b | Reflection time allocated during 
synchronous event 

2c | Reflection time allocated between 
events (multi-day) 

2d __| Small group discussions 

2e | Focus group discussions 

2f | Online role play 

2g | Self-assessment 

2h__| Online quizzes 

21 Practice 

3 Display 

3a__| Visual representations, such as: 
pictures, diagrams, graphs, multi- 
media, video, charts, animations 
3b__| Demonstrations 

4 Doing 

4a | Caselettes (short cases) 

4b__| Case studies 

4c Practice 

4d__| Project-based learning 

4e | Collaborative group project 

4f | Cognitive apprenticeship 

4g__| Assessment 

5 Coaching 

5a__| Cognitive apprenticeships 

5b__| Scaffolding 

5c __| Interactive feedback 

6 Conviviality 

6a | Ground Rules to encourage 
collaboration, cooperation, and trust 
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7 Critical Incident Technique (CIT) 
7a | Use CIT for feedback between multi- 
day instructional events 

8 Planning/Organization 

8a Fast Start 

8b | Quick Reference 

8c | Participant Manual 

8d | Timeline for all activities 

8e | Time for Reflection during events and 
between events 


Comments. Please use the space below to comment on any of the components of the model. Add any suggestions 
you have for elements that should be added to the model. 
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Appendix B 
MODEL EVALUATION SURVEY 2 


eSUCCESS Framework 


ESUCCESS Framework 
Please rate each element in the ESUCCESS Framework as to its importance in the model on a scale of | to 4. 1 is 
unimportant. 2 is of little importance. 3 is important. 4 is very important. 


Question 1 2 3 4 
1 eXecutive Sponsorship 
la Time 


lb Money 


le Resources 
ld Communications 
le Vision 


lf Change Management Strategy 
2 Support from the Organization 
2a Line of Business Executive 
2b Learning Executive 
2c IT Executive 
2d__| Direct Management Involvement 
2e | Project Manager for eLearning 
Initiatives 


2f Strong Technology Infrastructure 
2g __| Help Desk 
2h __| Built into the Organization 
2i Social Networks 
3 Display 
3a __| Visual representations, such as: 
pictures, diagrams, graphs, multi- 
media, video, charts, animations 
3b Demonstrations 
4 Understand and Motivate the 
Learner 
4a Benefits Communicated 
4b Included in Performance Plan 
4c Granted Additional Resources 
4d __| Continuing Education Requirements 
4e “Peer Pressure” 
4f | Mandated by the Regulators 
4g__| Pre-requisite to Live Training Event 
4h_ | Threat of Job Loss 
Ai Certification and Accreditations 


5 Culture Fosters Learning 
5a_ | Systems for Continuous Learning and 
Sharing 


5b__| Promote Dialogue and Inquiry 

5c__| Collective Mission 

5d__| Leader 

S5e__| Collaboration and Team Building 
5f | Connecting to the Environment 
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6 Content is Relevant to the Learner 
and Organizations 

6a__| Job and Task Specific 

6b | CEUs, PDUs, CPEs 

6c _| Certifications 

6d | Branded to the Culture and 
Environment 

6f | Recognized SMEs/Experts 
Facilitating/Teaching 

6g __| “Hot Topics” in the Industry 

7 Evaluate and Assess 

7a__| Kirkpatrick Level 1 

7b__| Kirkpatrick Level 2 

7c | Kirkpatrick Level 3 

7d__| Kirkpatrick Level 4 

8 Structure of Program is Engaging, 
Interactive, and Blended 

8a__| On-the-Job Training (OJT 


8b Coaching and Mentoring 
8c Web-based Technology 


8d Traditional Classroom 

9 Simulate the Work Environment 
and Work Tasks 

9a Goal-based Scenarios 

9b__| Problem-based Learning 

9c Case Studies 

9d_ | “Caselettes” 


Comments. Please use the space below to comment on any of the components of the model. Add any suggestions 
you have for elements that should be added to the model. 
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Appendix C 


Focus Group 
Nominal Group Technique-Model Validation 
eSUCCESS 
R2D2/C3PO 
Agenda 


Welcome and Overview (2 minutes) 

Process for each model 
Step 1. Review of Model Components and Silent Generation of question(s) to be addressed. What 
components might you add to each element in the model? And, to the model itself? (10 minutes) 


Step 2. Sharing Ideas. Each participant shares their ideas about additions. No debate at this stage (2-3 
minutes each). (10 minutes) 


Step 3. Group Discussion. (20 minutes) 


Step 4. Voting. (3) minutes 
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Appendix D 


Brookfield’s Discussion Techniques 


Circle of Voices 
Individuals reflect on the discussion topic (1-3 minutes) 
Participants go around the circle in order - each person has up to | minute of uninterrupted air time to 
give their viewpoint on the topic. No interruptions are allowed. 
Move into free discussion with the ground rule that every comment offered must somehow refer back 
to a comment made by someone else in the opening circle of voices. This need NOT be agreement - it 
can be a disagreement, a question, an elaboration or extension, an illustration, and so on. 


Retrieved from:http://www.stephenbrookfield.com/Dr._ Stephen _D. Brookfield/Workshop_ Materials. html 
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Descriptors: MOOCs, Graduate Supervision, Design and Development 


Introduction 


The Quality Graduate Supervision miniMOOC is an innovative design solution for faculty development 
that provides an accessible and flexible online learning environment for graduate supervisors. Many professors tend 
to develop graduate supervision skills on the job versus through active and intentional training. While many faculty 
welcome seminars on graduate supervision, the act of balancing demanding research, teaching and service roles and 
responsibilities can leave little discretionary time for scheduled faculty development seminars or workshops. To 
support both new and experienced graduate supervisors in continual professional learning to enhance their graduate 
supervision skills and knowledge, we designed a mini Massive Open Online Course (miniMOOC) on Quality 
Graduate Supervision at the University of Calgary (Alharbi & Jacobsen, 2016b). The first offering of the QGS 
miniMOOC is labelled “mini” because the first iteration is being offered as a closed pilot for research. 


This paper reports on the design and development phases in the production of the Quality Graduate 
Supervision miniMOOC. This innovative design solution is part of an overall design based research investigation 
into the design, implementation and evaluation of online faculty development for graduate supervisors. This paper 
builds upon the analysis and exploration phase of the minIMOOC (Alharbi & Jacobsen, 2016a) and describes the 
design elements that were determined to be relevant and necessary for the development of this online faculty 
development experience. Second, this paper describes the process and challenges experienced during the 
development of the QGS miniMOOC. Outcomes from the design based research project can inform the design and 
development of online faculty development and learning opportunities in higher education. Findings from this 
research are relevant for graduate students, academic faculty across disciplines, senior leadership in higher 
education, and educational developers. 


Background 


Graduate supervision is defined as “a complex pedagogical practice. It is a partnership between an 
experienced and an aspiring scholar, which shifts over the number of years it takes for the research to be done and 
the thesis to be written” (Kamler & Thomson, 2014, p. 1). Most graduate supervisors develop their practice 
experientially, that is, by just in time learning early in their academic careers (Alharbi & Jacobsen, 2016b). Some 
graduate supervisors approach graduate supervision in the ways that they were supervised, while others actively take 
steps to supervise students in different or better ways. It is clear that the quality of graduate supervision experienced 
by students varies (Beaudin, Emami, Palumbo & Tran, 2015). The relationship that develops between graduate 
supervisors and graduate students is one of the important factors in the success of students in their graduate studies 
(Adkins, 2009; Skarakis-Doyle & McIntyre, 2008). One issue in developing learning opportunities for graduate 
supervisors is the paucity of literature on how to best support graduate supervisors. A few pioneers in higher 
education are paying attention to the types of support and informal learning experiences that help to prepare 
graduate student for research and new faculty members for graduate supervision (see The Thesis Whisperer 
https://thesiswhisperer.com, and The Supervision Whisperers https://thesupervisionwhisperers. wordpress.com). 


38 


Research on student satisfaction with graduate supervision has been carried out (Beaudin, et al., 2015; Erichsen, 
Bolliger & Halupa, 2014), and advice is provided on helping graduate students with writing (Kamler & Thomson, 
2014) and with teaching (Altman, Stein & Stowell, 2015); however, there is a paucity of research on how to be a 
good supervisor beyond identifying some characteristics of good supervisors (Ismail, Hassan & Masek, 2014). 
Dangel and Tanguay (2014) wrote, "there is conceptual and empirical literature that points to the need for quality 
supervision; however, there is less information on how to best support supervisors in their work” (p. 4). 

On campus workshops, seminars and sessions focused on graduate supervision can also enhance the skills 
of newer and more experienced supervisors, and help them to develop expertise beyond their individual experiences 
being supervised. The busy schedule of faculty members, volume of work, logistical issues such as the times and 
locations of sessions, lack of recognition or financial awards for teaching, and lack of direction from the university 
are some of the reasons behind the lack of participation of faculty members in faculty development opportunities 
available at institutions (Taylor & McQuiggan, 2008). It is important to provide faculty development opportunities 
that are accessible, flexible and responsive to faculty members’ demanding work weeks. Furthermore, the design of 
a faculty development opportunity that focus on improving the skills of graduate supervisors should emphasis on the 
idea of creating a safe space for supervisors to discuss, critique and debate this private pedagogical relationship 
“supervision” (Manathunga, 2010). 

The use of MOOCs for faculty development offer one approach to addressing the accessibility and 
flexibility issues and thus may encourage greater participation among faculty members. Researchers have 
recommended the use of MOOCs as a workable solution for faculty professional development (Bond, 2013; Fini, 
2009). Fini (2009) explains that most people attend MOOCs for professional development reasons. Zhenghao et al. 
(2015) report that 52% of respondents who were surveyed after enrolling in one of Coursera’s MOOCs pointed out 
that the main reason they took a MOOC was to build their career by enhancing or improving their current jobs, or to 
find new ones. These studies indicate that MOOCs can provide accessible professional development opportunities 
and also have benefits for those who engage in the professional development. 

Furthermore, the use of MOOCs could be a solution for busy faculty members to participate in faculty 
development opportunities as they can determine the amount of time and effort they put into their learning and the 
time that better fit their schedule and it could help to blur the line between professional practice and professional 
development if the MOOC join communication, teaching and mentoring of faculty members (Bond, 2013). 

This research is guided by two assumptions: 1) that higher education can benefit from the use of MOOCs 
for faculty development, and 2) a MOOC can be designed that focuses on quality graduate supervision and creating 
a learning community among participants. 

This design based research is guided by three research questions. 
1. What design elements are necessary in the development of a Quality Graduate Supervision miniMOOC? 
2. What scaffolding and support is necessary to support faculty members to effectively use the Quality 

Graduate Supervision miniMOOC? 

3. In what ways can a Quality Graduate Supervision miniMOOC support and benefit faculty members in the 
ongoing development of their graduate supervision practices? 


Methodology 


The present research is a design based approach to designing, developing and evaluating a faculty 
development miniMOOC for quality graduate supervision. Wang and Hannafin (2005) define design-based research 
as “a systematic but flexible methodology aimed to improve educational practices through iterative analysis, design, 
development, and implementation, based on collaboration among researchers and practitioners in real-world 
settings, and leading to contextually-sensitive design principles and theories” (p. 6-7). This paper focuses on the 
development of an innovative solution, which is the second of four phases in this design-based research initiative: 
(1) analysis of practical problems by researchers and practitioners; (2) development of solutions; (3) evaluation and 
testing of solutions in practice; and (4) reflection to produce ‘design principles’ (McKenney & Reeves, 2012; 
Reeves, 2006). In this project, the development of an innovative solution for flexible and accessible faculty 
development was a six module, QGS miniMOOC, which was offered first as a closed pilot at a Canadian university 
to evaluate and test the solution in practice. The first offering of the QGS miniMOOC is labelled “mini” because this 
first iteration was being offered as a closed pilot for the initial evaluation phase of the research. 

Several factors, considerations and elements informed the design of the QGS miniMOOC. A conceptual 
framework for quality supervision emerged from a literature review and three learning theories: constructivism, 
connectivism and learning community. Principles of constructivism have been incorporated into the design to 
promote interactive learning, collaborative learning, facilitated learning, authentic learning, learner-centered 
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learning and high quality learning (Huang, 2002). Connectivism theory emphasizes that “instead of knowledge 
residing only in the mind of an individual, knowledge resides in the distributed manner across a network” (Siemens, 
2006, p.8). Therefore, the design of the miniMOOC emphasizes opportunities for knowledge exchange, discussion, 
networking and communication among faculty members from diverse disciplines and fields of study (Alharbi & 
Jacobsen, 2016b). Learning communities in higher education can emerge from the careful design of collaborative 
learning environments (Tennant, McMullen & Kaczynski, 2010). The development of a learning community in 
online faculty development can increase learner success; thus, the miniMOOC was designed to support faculty in 
engaging in and forming a learning community (Taylor & McQuiggan, 2008). The discussion board in the 
miniMOOC was designed to create a learning community among participants to discuss issues related to graduate 
supervision and to contribute to the ongoing discussions by giving advices and/or examples based on their expertise 
in graduate supervision along with the guidance of the discussion moderators. 


Process of the Design and Development of the miniMOOC 


Designing and offering professional development opportunities must be strongly tied to the institution’s 
strategic vision (Camblin & Steger, 2000). The design of the QGS miniMOOC is aligned with two goals of the 
university’s academic strategic plan: 1) to focus on innovative teaching and learning, 2) to promote the inter- 
institutional and inter-disciplinary collaboration and internationalization. The university is invested in supporting 
academic faculty members in developing strong graduate supervision practices and also recognizes excellence in 
graduate supervision through an established reward structure. The Faculty of Graduate Studies, University of 
Calgary, has established the “My Supervisor Skills” program to support graduate supervisors with various seminars 
and workshops throughout the year. Every new academic faculty member must complete an orientation seminar in 
order to be granted graduate supervision privileges. A faculty member’s supervisory privileges are evaluated every 
five years. Most faculties, including the Werklund School of Education, offer an annual Distinguished Graduate 
Supervision Award; the university offers the Teaching and Learning Award for Graduate Supervision each year; the 
Faculty of Graduate Studies grants GREAT Supervisor Awards each year during Great Supervisors Week; and the 
Graduate Students Association offers the Excellence in Supervision Award. An established reward structure is an 
important institutional component in the success of any faculty development as the lack of recognition from 
institutions may be a factor that discourages faculty from participating in faculty development (Taylor & 
McQuiggan, 2008). To that end, the support that the university offers for faculty members, along with visible 
recognition through awards for quality supervision, provided a strong foundation for the design and development of 
the Quality Graduate Supervision miniMOOC at the University of Calgary. 

The design process was also informed by a series of consultations with experts about elements of the 
design. In the analysis phase, questionnaires were sent to all graduate supervisors and graduate program directors at 
the University of Calgary to gather feedback about the challenges faced by graduate supervisors, the need for faculty 
development opportunities and on the desired topics that such faculty development should address (Alharbi & 
Jacobsen, 2016a). Key themes from the questionnaire data informed the development of the themes and the learning 
objectives for the six modules in the QGS miniMOOC. The six learning module themes and the sub-topics are listed 
in Table 1. 


Table 1. The six learning Modules of the QGS miniMOOC and Sub-Topics 


Week Topic Sub-Topics 
1 Introduction to Graduate - Introduction to the Quality Graduate Supervision miniMOOC, 
Supervision and supervision resources from the University of Calgary 
- Introduce the research and invite faculty to be involved. 
2 Best Supervision Practices - Elements of caring and effective supervision 


- Faculty of Graduate Studies guidelines for supervisor and 
student best practices 

- Supervision policy and regulations 

- Graduate program design and requirements 

- Supervisor-student checklist 


3 Relationship Building - How to establish good communication from the beginning 
- Creating shared expectations: 

-- Frequency of meetings 

-- Agenda and purpose for meetings 

-- Timelines and benchmarks 
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- What does the first meeting look like? 

- Supervision and mentoring across cultures 

4 Mentoring New Researchers - Supporting students’ coursework and program goals 

- Coaching and guiding through the various stages of the writing 
process, through grant applications 

- Strategies to motivate and support graduate students 

- What resources are available for graduate students? 


5 Anticipating and Addressing - Identifying students who are struggling 
Challenges - Conflict identification and prevention 
- Conflict resolution (what to do and where to go?) 
6 Promoting Excellence and Wellness | - Supporting students with knowledge mobilization and grant 
in Graduate Education writing 


- Developing research teams 

- Striving for balance 

- Supporting wellness and wellbeing in graduate students and 
faculty 

- Participant reflection on key learning and highlights 


Each of the six modules includes an overview, learning objectives, resources, videos and a discussion 
board. In the design and development phase, the first author finalized the design of each module with a faculty 
committee, and set to work developing each of the modules along with the expertise of members of the minIMOOC 
team. In addition to resources and guides available from the Faculty of Graduate Studies, a design decision was 
made to include videos of expert supervisors. One of the challenges at this stage of the research was to attract 
sufficient funding for the development the miniMOOC and in particular for high quality video production. 
Acknowledgement is given to the Faculty of Graduate Studies for providing a development grant for the QGS 
miniMOOC that enabled the team to work with a professional videographer and editor in the creation of the expert 
videos for the miniIMOOC. 


Questions were developed in relation to each of the six module themes and were used as a guide for the 
expert supervisors during the videotaped interviews. The expert supervisors were selected based on having won 
Faculty of Graduate Studies GREAT Supervisor awards and Faculty Distinguished Supervision Awards. The 
supervisors included three males and three females from such disciplines as psychology, biochemistry, medicine, 
education and nursing. The videographer set up a studio in a quiet office space to capture high quality video and 
sound. Using the prepared questions as a guide, the interviews with each expert supervisor enabled the design team 
to capture the diverse qualities and practices of excellent graduate supervision in relation to all six of the topics. 
Each video interview took approximately one hour. The source video was edited so that each of the final expert 
videos for each question in a module included perspectives from three different supervisors. The selection of the 
supervisors’ responses to each question was based on two considerations: 1) how informative was the answer, and 2) 
how diverse were the supervisor’s answers to the guided questions. The goal was to include composite expert videos 
for each module that presented diverse and rich responses to questions such as, what constitutes best practice in 
graduate supervision, how do you identify students who are struggling, and how do you motivate students? For 
example, in module three, a composite expert video on “how to establish good communication with your graduate 
students from the beginning” was posted along with two other composite expert videos related to this module’s 
topic. Another video on the course home page combines perspectives from all six expert supervisors on what is most 
satisfying about being a graduate supervisor (Figure 1). 
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Quality Graduate Supervision miniMOOC 


RSE MOME CONTENT CALENDAR 


Quality Graduate Supervision miniMOOC | 


To learn more about the QGS miniMOOC, click on 
“Content” in the menu above, then “Overview” 


To view the six week schedule, please click Weekly 
miniMOOC Schedule 


We asked each of our experts, “What is the most satisfying thing about being a graduate supervisor?” 


Access this video to hear what they had to say. 


—==m 


Savraj Grewal 
Associate Professor, 
Department of Biochemistry 
& Molecular Biology 


| > Fi = 


Figure 1. Home Page of the Quality Graduate Supervision miniMOOC 


Questions and prompts posted in each module’s discussion forum are aligned with the questions asked of 
the expert supervisors in the videos. In module five, the discussion board offers case scenarios that the participants 
were invited to discuss and debate with others. Four of the six experts’ supervisors who participated in the video 


interviews, and an Associate Dean from Faculty of Graduate Studies, were recruited to moderate the discussion 


forum for a week to answer questions and participate in the discussion. One discussion moderator requested the 
opportunity to guide the discussion for two weeks in a row after having a very informative experience in the first 
week. A summary of the design elements and components implemented in the QGS miniMOOC is in Table 2. 


Table 2. Design Components Implemented in the QGS miniMOOC 


Design Components 


Design and Construction 


Overview and Learning 
Objectives 


An introduction was developed for each topic of the QGS miniMOOC along with 
weekly learning objectives. 


Experts Supervisors’ 
Videos 


- A series of interview questions were developed in relation to each module 
topic to provide a guide for filming the expert supervisors. 
- Expert supervisors included three males and three females from such 


disciplines as psychology, biochemistry, medicine, education, counselling 
psychology and nursing. 


Discussion Board 


- Questions and prompts posted in each module’s discussion forums are aligned 
with the questions asked of the expert supervisors in the videos 

- Discussion moderators were involved in guiding, promoting and summarizing 
discussion each week 


Resources 


Resources and content were collected from the Faculty of Graduate Studies at the 
University of Calgary and from other online sources and was aligned with the 
weekly topics in the QGS miniMOOC. 
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The development process to ready the QGS miniMOOC for initial launch took approximately four months, 
including consultation with the miniMOOC team at every step of the design, gathering formative feedback on each 
module as well as the overall design and navigation, altering the design of the modules, consulting and getting more 
feedback, and making multiple changes and improvements through many iterations. The build using the learning 
management system took approximately two months. As part of the design and development phase, the QGS 
miniMOOC was accessed by three faculty members who were invited to serve as peer reviewers to evaluate the 
design, give feedback and suggestions before the miniMOOC was launched. Each of the reviewers who were 
recruited were selected for their recognized excellence in teaching and or in graduate supervision. 

The Quality Graduate Supervision miniMOOC was first offered as a closed pilot at one university in 
Winter 2017. A link to a recruitment video and email invitation to participate in the QGS miniMOOC was sent to all 
graduate supervisors at the University of Calgary. Diverse faculty members from computer science, education, 
political science, medicine, veterinary medicine, sociology, art, engineering, physics & astronomy, nursing and 
physiology & pharmacology chose to take part in the first offering of the QGS miniMOOC (Alharbi & Jacobsen, 
2017). The QGS miniMOOC was launched via three orientation sessions (online and on-campus) for the 23 
graduate supervisors from eleven different disciplines who initially signed up. The purpose of the orientation was to 
introduce the faculty members to the QGS miniMOOC, provide a brief demo, and introduce them to the study. Each 
supervisor was provided with a binder that included an overview of the six modules, the FGS Supervisor Handbook 
produced by the Faculty of Graduate Studies, a copy of the orientation slides, and several printed resources from the 
first two modules. Supervisors were encouraged to print and save resources in their binder that they found to be 
helpful from the QGS miniMOOC. A demo was provided to show faculty members how to login the minIMOOC 
and navigate the different modules. An online seminar was provided for faculty members who could not attend in 
person. Graduate supervisor participants were unanimously very enthusiastic during the orientation about the QGS 
miniMOOC learning experience and were eager to get started — more than half viewed the videos, accessed the rich 
resources and engaged in the discussion forums during the first few days that the QGS miniMOOC was open. 

The instructor sent the 23 graduate supervisor participants a group email at the beginning of each week to 
guide them on how to engage in the minIMOOC activities and with the learning resources (see Table 3). 


Table 3. Example of Instructor’s Weekly Email to miniMOOC Participants 
Good Morning Graduate Supervisors!! Welcome to Module 4 and Week 4 in the Quality Graduate Supervision 
miniMOOC! Our topic this week is Mentoring New Researchers. 


Module 4, Mentoring New Researchers: This module has several objectives: to gather ideas about how to 
support students with coursework and program goals, to access strategies for coaching and guiding students 
through the writing process, to learn strategies from others about how to motivate students to make timely 
progress in their programs, and to access the plethora of resources available to support students and 

supervisors. For example, I recently learned about this treasure trove of helpful resources by Canadian and UK 
researchers: http://doc-work.mcgill.ca I will be adding this reference to Module 4, and also accessing it heavily 
for my own graduate supervision. 


Keeping yourself on track: Academic faculty are organized, right? The value of online learning is 

the accessibility and flexibility to make it work for YOUR busy schedule! However, it works best when you 
_schedule_ MOOC learning into your week! If you have not done so already, I encourage you to add 
"miniMOOC" to your calendar, and login several times per week to access resources, watch the videos, and to 
engage with others in the discussion forum. 


Learner Expectations for this module: As with the first three modules, you are expected to: 1) Access and 
review the rich resources, 2) Watch the expert videos (and take jot notes!), 3) Engage in the Community of 
Practice by contributing to the Discussion Forum, and 4) Respond to discussion contributions made by 
others. Optional Portfolio - you can complete any activities and add useful resources you have found to your 
FGS Supervision Handbook. 


Discussion Forum & Moderator: I am thrilled to introduce the Discussion Moderator for this week, Dr. Name 
Professor, Department Name. Many of you may feel you already know Dr. Name from the Expert Videos! Dr. 
Name will be on hand this week to answer questions and w will also be taking part in the discussion forum! 


I will be checking in with our group regularly, either through email or by taking part in the discussion forum. 
All the best in continuing your active learning journey in the miniMOOC. 
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Of the 23 faculty members who signed up for the Quality Graduate Supervision miniMOOC, a total of 15 
graduate supervisors completed all six of the modules and were presented with a certificate of completion. Of the 15 
faculty members who completed, 2 had requested and benefitted from extra time beyond the six weeks to complete 
the final modules. As for timing, the pilot was offered in March and April, and the final two modules overlapped 
with the end of semester grading and reporting period. In future offerings, the development team will offer the 
miniMOOC earlier in the semester, like October and November, or February and March, or even in Spring or 
Summer semesters, so that the final two modules do not overlap with the demands of end of semester. 

As part of the overall design based research project, data was collected using interviews, surveys and 
observations with participants during and at the end of the course to evaluate the implementation phase, to learn 
about what worked and what needs to be changed, and to inform the design principles for faculty development 
MOOCs. While data is currently being analyzed, and full research findings from the entire design based research 
project will be shared in a doctoral dissertation and subsequent presentations and publications, it is fair to conclude 
that the QGS miniMOOC pilot was a successful learning experience for faculty participants and that this project will 
continue to expand to a wider audience at the University of Calgary, and eventually, beyond our campus to graduate 
supervisors from the wider higher education community. 
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Abstract 


A review of literature was conducted to determine effective diversity training practices in organizations for 
change leaders as well as the ethical implications of completing such training. The research resulted in an updated 
definition of diversity; depicted factors that impact the effectiveness of diversity training, and provided specific 
strategies and tactics that can be used for effective diversity training in organizations. Research showed that 
effective and successful diversity training focuses on ethical treatment of diverse individuals while minimizing and 
addressing resistance, through tactics such as coaching and staffing, as well as promoting training practices through 
open communication. Diversity training includes strategies to address ethics, including: 1) a workplace atmosphere 
of harmony and democratic citizenship, 2) acceptance of differences and civility toward one another, and 3) using 
and valuing diversity attributes to benefit the individual and the organization. An ethical framework for 
implementing diversity training was designed around those three strategies and includes such tactics as coaching, 
intercultural competency training, migration management, workplace management, and open communications. 


Diversity Training in Organization Settings: 
Effective and Ethical Approaches for Change Leaders 


Diverse workplaces require leadership practices that meet the needs of a diverse population (Byeong, 
2006). As both the definition of diversity and the number of diverse individuals in the workforce expand, diverse 
organizations have grown to encompass all organizations (de Jong, 2016; Derderian-Aghajanian, 2010; Molina- 
Giron, 2016). Diversity training initiatives can increase the efficiency and effectiveness of organizations while also 
increasing employee satisfaction and civil treatment through the development and implementation of diversity 
training to create a culture of diversity understanding (Banks, 2007; Byeong, 2006; de Jong, 2016; Guerra, 2012; 
Molina- Girén, 2016; Von Bergen & Collier, 2013). 

Diversity training should incorporate such strategies as: the facilitation of acceptance and civility; value 
and utilization of diversity attributes, and the creation of a harmonic workplace atmosphere that facilitates an 
effective workplace with ethical treatment (Banks, 2007; Blum, 2007; Byeong, 2006; de Jong, 2016; Molina- Giron, 
2016; Von Bergen & Collier, 2013). These three strategies, and five specific tactics for implementing them, are 
shared in this research paper. The strategies and tactics for diversity training are shared in the form of a diversity 
training implementation framework, which is used to create a workplace culture of diversity understanding that 
considers ethical implications for the individual as well as the organization. 
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Statement of the Problem 
Purpose 


Diversity in United States workforce is increasing with people from different regions, races, socioeconomic 
status, religion, gender, sexual orientation, and other person identifiers entering and becoming more active in the 
workplace (Blum, 2007; Byeong, 2006; De Leon, 2014; Guerra, 2012). According to Guerra (2012), the application 
of outdated diversity approaches (e.g. archaic diversity training, equal employment opportunity practices, training to 
modify negative workplace behavior and hiring for diversity in traditionally-marginalized groups) do not fully meet 
the needs of diverse individuals. This idea is further supported by Blum (2007) and Coates (2014), who provide 
examples of the inability for current diversity approaches to effectively meet the needs of all people. Through 
current diversity approaches, both employees and organizations are not fully considered (Blum, 2007; Guerra, 2012; 
Von Bergen & Collier, 2013). Diverse workplaces and their respective personnel are provided with inadequate 
training as well as diversity changes and improvement practices that cause employees to feel uncomfortable and 
unvalued in organizations as well as within society (Blum, 2007; Byeong, 2006; Coates, 2014; Guerra, 2012; Von 
Bergen & Collier, 2013). 


Definition of Diversity 


For the purposes of this research, diversity is defined as any difference that exists within a person that can 
be classified by gender, race, sexual orientation, ethnicity, culture, language, religion, socioeconomic status, 
geographic location, mental and physical disabilities, political affiliation, subculture, and any other personal 
identifiers (de Jong, 2016; Derderian-Aghajanian, 2010; Fleming, Ledbetter, Williams, & McCain, 2008; Molina- 
Giron, 2016; Marshall & Theoharis, 2007). The research team of Fleming et al. (2008), shaped this definition 
through their research, which surveyed engineering students from four different universities; these students defined 
diversity as “gender roles and identity, and racial identity” as well as “gender, race, culture, and ideology” and, 
though less common, “major [academic areas of study], geography, socio-economic status, and political affiliation” 
(Fleming et al., 2008, p. 2). 

Other researchers shaped the definition of diversity by offering more traditional examples of differences. de 
Jong discovered that diversity includes aspects such as age, gender, sexuality, ethnicity, and migration origin 
locations (2016). Molina-Giron (2016), shaping the definition of diversity to include traditionally-marginalized 
groups, defines diversity as any occurrence where a person (or people) is “racially, ethnically, culturally, and 
linguistically diverse” (p. 142). Derderian-Aghajanian (2010) contributed to this definition by defining diversity as 
minorities and those with culturally diverse backgrounds. Marshall and Theoharis (2007), another research team 
adding to this definition of diversity, found diversity to be based on “race, social class, religion, ability/disability, or 
sexual orientation” (p. 1). Considering these different definitions of diversity and the variety of aspects that diversity 
is noted to encompass, a new definition of diversity is created. This new definition implies that all people are diverse 
and, therefore, all organizations are diverse organizations. All people, then, should value and emulate diversity 
understanding, as all are directly impacted by diversity practices (Banks, 2007; Parker, 2003; Patrick & Kumar, 
2012). 


Need for Effective and Ethical Diversity Training 


There is a need to obtain effective approaches to diversity training within organizations to enable change 
leaders to reach diverse personnel and ensure that they feel understood, valued, and accepted (Von Bergen & 
Collier, 2013) while also “protect[ing] and nurture[ing] cultural pluralism and equality (Parker, 2003, p. xvii). 
Considering this need, the purpose of diversity training should be to: 1) discover effective ways for leaders to 
implement diversity training in organizations, 2) consider the ethical implications of diversity training and diverse 
workplace environments on individuals, the workplace and society 3) create a framework for the implementation of 
effective diversity training that considers individuals, the organization and the society to which the individuals 
belong. 

Diversity is on the rise in the United States (Pope, 2012; De Leon, 2014). Understanding different people 
and the ways in which they interact is an important part of being a citizen in the United States, as “to live and 
function in increasingly multicultural, democratic societies” has become necessary (Molina-Giron, 2016, p. 143). 
Professionals “today acknowledge the importance of intercultural competence in conducting work ethically and 
efficiently, especially when serving racial minorities and diverse populations” (De Leon, 2014, p. 17). Due to 
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increased diversity and the additional aspect of the onset of the Tech Century and Society 3.0, which facilitate 
globalization and a world-wide workforce within organizations (Moravec, 2013; van den Hoff, 2014), organizations 
and the people working within them are called upon to be global citizens and function effectively and ethically in 
diverse settings with diverse cultures around the world (Bresselink, 2013; De Leon, 2014; Moravec, 2013). 
Additionally, an influx of people entering the workforce of the United States hailing from diverse cultures and 
locations requires organizations and their employees to understand and appreciate the uniqueness of every employee 
while functioning as a group to accomplish organizational objectives (Byeong, 2006; De Leon, 2014). 

Since the 1960s, diversity acceptance has become a key political and social issue (Blum, 2007; Coates, 
2014; Gorski, 1999). Though some improvements have been made for some groups of diverse people (e.g. same sex 
marriage, freedom to practice varying religions, the inclusion of women in male-dominated industries, and 
interracial schools), there is still a sizable gap between our current status and truly accepting diverse people (Blum, 
2007; Coates, 2014; Molina-Giron, 2016). The needs of traditionally-diverse (e.g. traditionally-marginalized) 
individuals should be treated with equal consideration as those of individuals who are not traditionally-diverse 
(Blum, 2007). Social justice must be universally applied but not “encased in standards or slogans that make one feel 
good but take away the responsibility of doing what is right” (Gutek, 2014, p. 126). 

Diversity in the workplace is increasing (Byeong, 2006; De Leon, 2014; Guerra, 2012). Fleming et al. 
(2008) found that there are “more women and minorities in the field” than there have been in the past, and 
employers are looking for effective “ways to attract and retain” them (p. 1). Professionals must be “able to relate to 
many cultures with the knowledge, skills, and attitudes” to effectively work with and understand diverse people 
(Derderian-Aghajanian, 2010, p. 154). A lack of diversity understanding leads to issues for diverse people, 
particularly in career development, with challenges that include communication and understanding, discrimination, 
cultural barriers, religious barriers, and access to resources (Blum, 2007; Coates, 2014; Pope, 2012). 

Organizational diversity is an issue that is still not readily addressed (Guerra, 2012; United States Equal 
Employment Opportunity Commission, 2017; Von Bergen & Collier, 2013). Most of the organizations that employ 
diversity training and attempt to meet diverse employee needs do so with equal employment opportunity (EEO) laws 
and regulations that do not effectively provide diversity training or resources for diverse employees (Guerra, 2012; 
Molina-Giron, 2016). While no specific EEO claims have significantly lessened over the last five years, race and 
gender-based EEO claims have gone down slightly, with approximately 1,000 less claims and 3,000 less claims, 
respectively (United States Equal Employment Opportunity Commission, 2017). Other claims, such as retaliation, 
equal pay, and disability, have shown an increase in charges (United States Equal Employment Opportunity 
Commission, 2017). This shows that though EEO is in place within organizations, it is not alleviating diversity- 
related negative claims in organizations, according to the charge statistics provided by the Unites States Equal 
Employment Opportunity Commission (2017). Additionally, workplaces with diverse employees and organizations 
that work with global clientele currently strive to train employees with the goal of diversity tolerance or validation 
(which was found to be inadequate) rather than acceptance and civility (Von Bergen & Collier, 2013). 


Methods 
Research Method 


A review of literature is the research method employed. Literature form a variety of sources was read and 
analyzed pertaining to diversity, ethics of diversity training, treatment of diverse people, and effective diversity 
training methods for change leaders. The review of the literature will be critically analyzed to create a framework for 
the effective implementation of diversity training in organizations. 


Research Questions 


The researcher focused on strategies and tactics for diversity training in organizations. Additionally, the 
researcher considered the ethical implications of such practices as well as the impacts of diversity training on 
organization personnel. Two primary questions shaped this research: 

1. What specific leadership strategies and tactics are effective for diversity training in organizations? 
2. What ethical implications should be considered by change leaders in diversity training? 
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Research Foundations 


A review of literature served as the method for this research. A qualitative analysis of philosophical ideas 
and theoretical foundations within both anecdotal and social research settings provided the data. 

The existentialist educational philosophy was applied to this research. Existentialists do not consider there 
to be an “ultimate reality” or “structure of the universe” (Gutek, 2014, p. 105); rather, existentialist educational 
philosophers consider reality to be subjective and individually-created based on “the perspective of the individual 
human person” (Gutek, 2014, p. 106). Existentialists are inclusive and do not consider individual attributes to be 
right or wrong (Gutek, 2014). The philosophy values individual choice and acceptance of people’s choices free from 
judgement and focused on social justice and inclusiveness (Gutek, 2014). Education and training, from an 
existentialist perspective, does not lead to the presentation of a correct or incorrect way in which to live (Gutek, 
2014), but existentialism does value education and training as a “way to examine life from multiple perspectives and 
to use these perspectives to raise consciousness about our [humanity’s] situation” (Gutek, 2014, p. 117). This is 
directly applicable to diversity training in organizations, as it, too should seek to provide views on differences and 
promote an idea of acceptance and civility with a raised consciousness rather than a judgement of what is right or 
wrong, good or bad, and correct or incorrect (Byeong, 2006; de Jong, 2016; Molina-Giron, 2016; Von Bergen & 
Collier, 2013). 

While no educational theory directly meets the needs of diversity training, there are aspects of critical 
theory that meet some requirements. Diversity training, with its required understanding of the globalization of 
different businesses and the influx of employees from different parts of the world (Bresselink, 2013; De Leon, 2014; 
Moravec, 2013), and critical theory, with its negative judgment toward globalized workforces and neo-liberal 
globalization ideals, do not align with one another (Gutek, 2014). However, other aspects of critical theory do align 
with diversity training in organization settings. The lack of exploitation, equality for all people, raised consciousness 
of individuals, social goals of education, respect for all people, and encouragement of individual beliefs are all 
valuable aspects of diversity training (Blum, 2007; Byeong, 2006; de Jong, 2016; Derderian-Aghajanian, 2010; 
Gutek, 2014; Hansen, 2007; Molina-Giron, 2016; Von Bergen & Collier, 2013). Critical theory emphasizes 
individual choice from “the results of interface and sharing of people whose voices have an equal right to be heard” 
rather than “imposition from those in power” (Gutek, 2014, p. 440). The critical theory values of equality, 
understanding, respect, dialogue (e.g. communications), lack of exploitation, sharing of opinions, and education to 
create social justice all align with diversity training (Byeong, 2006; de Jong, 2016; Gutek, 2014; Molina-Giron, 
2016; Von Bergen & Collier, 2013). Globalization is the area in which critical theory does not fully address the 
needs of diversity training (Byeong, 2006; de Jong, 2016; Gutek, 2014; Molina-Giron, 2016; Moravec, 2013; Von 
Bergen & Collier, 2013). 

Diversity training for globalized organizations is considered under the neo-liberal educational ideology. 
Neo-liberalism, a modern ideology, considers organization-based markets on a global scale (Gutek, 2014). 
Globalization, with international organization markets and an international economy, occurs in the twenty-first 
century and includes such aspects as: a global market, global entertainment, global information and communication, 
and global education (Gutek, 2014, Moravec, 2013; van den Hoff, 2014). Organizations today are both diverse and 
globalized (Moravec, 2013; van den Hoff, 2014), and an ideological understanding of globalization and its 
economic, educational, communications, and entertainment aspects (Gutek, 2014) allows diverse organizations to 
embrace globalized workforces and better meet organizational and personnel needs (van den Hoff, 2014). Whether 
personnel require cultural training, technology training, or other personal and professional development, diversity 
requirements must be both considered and met to facilitate effective globalized organizational practices (Moravec, 
2013; van den Hoff, 2014), including diversity training for globalized workforces. 

The authors would like to acknowledge personal and professional bias that may impact this research. The 
question of effective, ethical diversity training was approached with an existentialist educational philosophy at its 
foundation of beliefs. Existentialists believe that everyone will continually shape their own ideas of reality, no 
specific approach to life is “right” or “wrong,” ethics should always be considered, and all people (both outside and 
within traditionally-marginalized populations) deserve respectful, equal, civic treatment (Banks, 2007; Gutek, 2014). 
A neo-liberal ideology also influences the data analysis and strategies and tactics selected for the diversity training. 
Neo-liberal approaches consider the world on a global market level (Gutek, 2014) and the ways in which diversity 
impacts global organizations and their respective, diverse employees. 
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Review of Literature 


The review of literature showed that diversity training can be effectively accomplished by change leaders 
with the implementation of three strategies: creation of a foundation of acceptance and civility (Blum, 2007; 
Hansen, 2007; Von Bergen & Collier, 2013), the utilization and value of diversity (Blum, 2007; Molina-Giron, 
2016), and the fostering of a harmonic atmosphere (Byeong, 2006). These strategies can be implemented effectively 
through such tactics as coaching and mentoring (Hansen, 2007; Yirci, Karakose, & Kocabas, 2016), effective and 
open communications (Blum, 2007; Thakur & Thakur, 2008), staffing and intercultural competency development 
(Dimitrov, Dawson, Olsen, & Meadows 2014); migration management and retention (de Jong, 2016), and workplace 
management (Byeong, 2006). The strategies and tactics align with the ethics present within the existentialist 
education philosophy, critical theory of educational philosophy, and neo-liberal ideology (Gutek; 2014). 


Strategies 


The strategies of acceptance and civility, valuing diversity, and creating a harmonic workplace atmosphere 
alleviate obstacles and promote effective diversity training and, therefore, understanding within diverse 
organizations (Banks, 2007; Blum, 2007; Byeong, 2006; de Jong, 2016; Molina-Girén, 2016; Von Bergen & Collier, 
2013). The strategies discovered emphasize civil behavior, global applications, acceptance of differences, and just 
and equal treatment of individuals (Byeong, 2006; de Jong, 2016; Guerra, 2012; Gutek, 2014; Molina-Giron, 2016; 
Von Bergen & Collier, 2013). Each of these strategies considers social justice, equal treatment and acceptance of 
difference, and globalized diversity considerations, as do existentialism, critical theory of education, and neo-liberal 
ideologies, respectively (Gutek, 2014). 

A literary review of research on diversity training revealed that organizations and their leaders must work 
to foster environments of acceptance and civility (Blum; 2007; Von Bergen & Collier, 2013). Diversity interactions 
are described by Von Bergen and Collier (2013) in three ways: tolerance, acceptance, and validation. While Von 
Bergen and Collier view tolerance as asking too little of employees (it creates negative feelings and forces people to 
endure one another) and validation as asking too much (it creates negative emotions as people feel as though they 
are forced to resign their own values and personal beliefs), the authors view acceptance as the ideal middle ground, 
where employees can value one another without negative feelings of endurance or sacrifice. Acceptance of 
differences is coupled with respectful, civil, equal treatment of one another (like that depicted within existentialism 
and critical theory of education) to build the foundation of appropriate workplace interactions among personnel 
regardless of differences (Gutek, 2014; Von Bergen & Collier, 2013). Von Bergen and Collier (2013) reframe 
positive workplace behavior in diverse settings as acceptance and civility, noting that it increases workplace 
efficiency and learning while creating a foundation of respect, self-regulation, and appropriate public behavior. 

Understanding diversity is valuable to all people, whether they are traditionally-marginalized and diverse or 
not, and applying diversity training to facilitate ethical, equal, respectful, civil, accepting, valuing behavior toward 
all people regardless of differences should be a priority for everyone (Banks, 2007; Blum, 2007; Byeong, 2006; de 
Jong, 2016; Molina-Giron, 2016; Patrick & Kumar, 2012; Von Bergen & Collier, 2013). Molina-Giron (2016) 
details the importance of understanding diversity and using differences to promote success with organizational 
changes. Banks (2007), Parker (2003), and Molina-Giron (2016) stress the importance of balancing unity and 
diversity while promoting democratic citizenship among employees, regardless of their born cultures and geographic 
locations. This idea is present within existentialist thinking as well as both critical theory and neo-liberal 
globalization ideals (Gutek, 2014). Molina-Girdn (2016) notes that an environment of total unity can make diverse 
employee qualities less apparent and less utilized with individuals in the workplace (e.g. language proficiency, 
amount of eye contact given to others, competency and awards recognition, empathy, etc.); it is important to 
continue valuing individual diversity while functioning as a unit in order to be successful in diverse workplaces 
(Banks, 2007; Molina-Giroén, 2016). This enables diversity in the local and global workplace to be harnessed, as it is 
valued and celebrated (Blum, 2007; Byeong, 2006; Gutek, 2014; Molina-Girén, 2016). The utilization and value of 
diversity helps leaders create efficient workplaces in which positive changes can occur and diversity, along with 
personnel, can flourish (Molina-Giron, 2016; Patrick & Kumar, 2012). 

A continued review of literature revealed that organizations take one of three approaches to diversity 
(Byeong, 2006); they either have diversity, manage diversity, or value diversity (Byeong, 2006). While managing 
and having diversity are helpful when implementing diversity training, valuing diversity, such as acceptance of 
difference and a willingness to understand others (e.g. existentialism and critical theory), is where organizations will 
see continued success and growth as well as more productive employees and better workplace environments 
(Byeong, 2006; Gutek, 2014). This continued value of individual diversity helps leaders promote a collective effort 
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in order to successfully implement changes and foster an existentialist-based harmonic workplace atmosphere 
(Blum, 2007; Byeong, 2006; Gutek, 2014). Merged with a foundation of acceptance and civility in addition to the 
employment of diversity and democratic organization citizenship, an atmosphere of harmony, and the value of 
diversity helps leaders promote effective diverse workplace changes (Blum, 2007; Byeong, 2006; Gutek, 2014). 
Each of these actions, or strategies, are directly reflective of existentialist, critical theory and neo-liberal market 
values (Gutek, 2014), exemplifying their ability to reach all people and meet their respective, diverse needs 
(Byeong, 2006; de Jong, 2016; Guerra, 2012; Gutek, 2014; Molina-Giron, 2016; Von Bergen & Collier, 2013). 


Tactics 


The review of literature provides five specific tactics for effective diversity training. When implemented 
with the above-mentioned three strategies, these tactics (coaching, communications, intercultural competency 
development, migration management, and workplace management) create effective diversity training within 
organizations. These five tactics, like the three strategies defined above, align with the values of ethical treatment, 
respect, civil behavior, globalized markets, understanding, and freedom, which are expressed through existentialist, 
critical theory, and neo-liberal ethics (Gutek, 2014). 

According to Cohen (2005) and Kotter (2002), coaching works to promote employee proficiency and 
development within a person (internal or external) by encouraging employees and providing a first-hand, one-on-one 
or small group demonstrations of best practices within a given area (e.g. diversity training). Yirci, Karakose, and 
Kocabas (2016) address such aspects as the benefits of coaching, impacts on job performance, principles of practice, 
and the overarching purpose of coaching for training purposes. Yirci, Karakose, and Kocabas (2016) stated that 
these positive effects are not possible without addressing cultural needs and diversity within organization settings. 
Consideration and adaptation of practices into the cultural context while addressing diversity is an important factor 
in training success (Yirci, Karakose, & Kocabas, 2016). Additionally, Hansen (2007) notes the importance of 
coaching regarding moral development and understanding, depicting the existentialist and critical theory values of 
acceptance and individual freedom (Gutek, 2014). Coaching, mentoring, and attentiveness, according to Hansen 
(2007), help teach morals to students and assists in their ability to appropriately “respond to and treat one another, 
especially with regard to one another’s ideas and endeavors” (p. 353), thus promoting understanding of diversity. 

Communication tactics, such as those noted by Thakur and Thakur (2008) attribute success to cultural and 
diversity training. Communication, in general, is successful within diverse workplaces and organizations that 
conduct business with diverse clients, when personnel are trained on diversity and diversity is considered and 
understood (Thakur & Thakur, 2008). Communication between different cultures has a large impact on 
organizational networking, communication, foreign activities, and psychology (Thakur & Thakur, 2008), and a 
thorough understanding of the diverse persons involved with the organization in any way should be attained prior to 
communicating with them (Thakur & Thakur, 2008). Due to the possibility of distorted communication between 
diverse people, and the importance of communicating across the global marketplace, this understanding is essential 
for both respectful and accurate transmission of thoughts, materials, content, and other items (Gutek, 2014; Thakur 
& Thakur, 2008). Ultimately, Thakur and Thakur (2008) determined that organizations must consider differences in 
people, including their cultures and their communication practices, to effectively communicate across diverse planes 
and global business markets, such as those expressed in the neo-liberal ideologies (Gutek, 2014). 

Dimitrov et al. (2014) note intercultural competence development and staffing as tactics that are effective 
when diversity exists within a workplace. Competency development is a form of training through human resources 
that views employee differences and educates employees to better conduct business across multiple cultures 
(Dimitrov et al., 2014). Aspects of existentialism and critical theory ethics, as expressed by Gutek (2014), are 
emphasized in diversity competency development as noted by Dimitrov et al. (2014). These skills include 
communication, relationship development, tolerance, openness, understanding, positive attitude, cultural knowledge, 
culturally-appropriate strategies, and the ability to model these actions (Dimitrov et al., 2014; Gutek, 2014). 
International and intercultural skills are essential for success in the workplace, as is an inclusive environment that is 
free from judgement (Dimitrov et al.,2014). Skills such as increased cultural awareness, understanding of cultural 
limitations, awareness of contextual differences, and avoidance of misunderstandings are successfully developed 
with the consideration of diversity through intercultural competence training (Dimitrov et al., 2014). Additionally, 
appropriately staffing individuals and placing them in positions in which they can flourish helps facilitate diversity 
training and competency development while also promoting individual employee growth and success (de Jong, 
2016; Dimitrov et al., 2014). 

Migration management and employee retention also attribute to the success to diversity training in the 
workplace (de Jong, 2016). These tactics, as noted by de Jong (2016), focuses on approaches for handling increased 
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migration from geographically-diverse employees into an organization while also providing a better perspective on 
managing a diverse workforce. It considered the global marketplace as well as valuing individuals for their 
differences and providing personnel with the freedom to be themselves and value for doing so (de Jong, 2016; 
Gutek, 2014). Migration management and diverse employee retention work to obtain a triple-win (value for 
employers, employees, and customers), as noted by de Jong (2016). Focuses on flexibility, proactive decisions, 
phenomena management, depoliticization, international partnership understanding, diversity management, risk 
protection, and increased expertise and knowledge are all part of this human resources tactic (de Jong, 2016). de 
Jong (2016) notes that organizations must synthesize migration and diversity management into an overarching 
program that directly address diversity training needs and puts the right person in the right place (e.g. staffing). 
Workplace management, as noted by Byeong (2006), addresses diversity and helps employers and 
employees to understand, value, and harness diversity to become successful organizations and create an environment 
of efficiency and effectiveness. Since workplaces are constantly changing, an understanding of different cultures and 
diverse perspectives allows organizations to achieve change success while also employing personnel to their fullest 
potential (de Jong, 2006). Valuing and understanding diversity are essential to diverse organization success, 
according to de Jong (2016), and simply having and managing diversity is not enough (de Jong, 2006). Diversity 
must be valued and understood in an accepting environment where individuals are free to be themselves while 
respecting each other’s differences, noted ethical aspects of existentialism and critical theory (Gutek, 2014), to 
obtain such benefits as fresh ideas, skill development, competence, flexibility, responsiveness, enthusiasm for 
change, growth, enhanced image, and the gain of new, valuable employees (de Jong, 2006). de Jong (2006) found 
that diversity training, understanding, and adaptation are important to the success of diverse workplace management. 


Framework 
Diversity Training Framework 


The findings of the review of literature depicted leadership strategies and tactics for successful diversity 
training implementation in workplaces that consider ethical implications and practices based on existentialism, 
critical theory of education and neo-liberal globalized marketplaces (Gutek, 2014). From these strategies, tactics and 
ethical considerations, a training framework, called the Diversity Training Framework, was created. The function of 
the Diversity Training Framework is to facilitate effective diversity training in organizations while actively 
considering ethics and individual values and freedoms. The ethical framework is based on a foundation of 
acceptance and civility, a workplace atmosphere of harmony and democratic citizenship, and the value and use of 
diverse attributes employees possess to benefit each employee and the organization (Byeong, 2006; de Jong, 2016; 
Gutek, 2014; Molina-Giron, 2016; Von Bergen & Collier, 2013). 

The Diversity Training Framework is depicted as a pyramid. When implemented by change leaders, the 
three strategies (forming the base of the pyramid) and five tactics (forming the top layers of the pyramid) complete 
the Diversity Training Framework while considering ethics (e.g. those aligned with existentialist, critical theory, and 
neo-liberal values). 

The framework has six levels: 

1. Foundation (acceptance and civility; atmosphere of harmony; valuing diversity) 

2. Open Communications (dialogue; communication of values and ideas; sharing of mission, goals 
and purpose) 

3. Intercultural Competency Development (relationships; multi-cultural interactions; value different 
ideas and abilities) 

4. Migration Management (value for employees, employers and clients; proactive decisions; 


flexibility) 

5. Workplace Management (staffing; value individual diversity; enable fullest potential for 
employees) 

6. Coaching (individual attention and motivation; moral guidance; best practices and effective 
methods) 
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Figure 1. Diversity Training Framework. 


Analysis and Ethical Implications 


Change leaders should understand diversity and apply ethical diversity training practices that reach beyond 
what is seen on the surface, delving into differences that are not immediately visible (Banks, 2007; Guerra, 2012; 
Gutek, 2014; Molina-Giron, 2016). Differences effect the entire person (Gutek, 2014), and understanding diverse 
things such as such eye contact, humility, sharing of accomplishments, motivation, ethics, and other hidden factors 
are necessary to successfully implement ethical diversity training in diverse workplaces and promote diversity 
acceptance (Guerra, 2012; Gutek, 2014; Molina-Giron, 2016; Von Bergen & Collier, 2013). 

Research shows that ethical strategies and tactics, as well as ethical individual considerations, effectively 
facilitate diversity training and understanding, resulting in more effective and ethical workplaces (Byeong, 2006; de 
Jong, 2016; Gutek, 2014; Molina-Giron, 2016; Von Bergen & Collier, 2013). The Diversity Training Framework, 
which focuses on: creating an atmosphere of harmony; facilitating acceptance and civility; valuing diversity, and 
implementing such practices as coaching, communication, migration management, workplace management, and 
intercultural competency development, can facilitate ethical and effective diversity training and create a more 
effective diverse workplace (Byeong, 2006; de Jong, 2016; Dimitrov et al., 2014; Gutek, 2014; Thakur and Thakur, 
2008; Yirci, Karakose, & Kocabas, 2016). When used together, each of the strategies and tactics within the 
Diversity Training Framework enable effective diversity training that addresses the ethical (as represented by 
existentialist, critical theory, and neo-liberal values) needs of employees and the organization, including civility, 
acceptance, harmony, democratic citizenship, respect, and equal, appropriate public behavior (Byeong, 2006; de 
Jong, 2016; Derderian-Aghajanian, 2010; Guerra, 2012; Gutek, 2014; Molina-Giron, 2016; Von Bergen & Collier, 
2013). 

Existentialist ethical values, such as individual experiences and a contextual, ever-evolving reality that calls 
for acceptance and a lack of judgment (Gutek, 2014), are illustrated in the Diversity Training Framework. Strategies 
that focus on acceptance, civil treatment, harmonic workplaces, and valuing individual abilities and diversity factors 
(Bank, 2007; Byeong, 2006, de Jong, 2016; Molina-Giron, 2016; Von Bergen & Collier, 2013) are directly 
reflective of existentialist ideals, such as those shared by Gutek (2014). They align with the ideas of multiple 
perspectives and raised consciousness through knowledge of differences and understanding as well as a lack of 
“right” or “wrong” judgment of differences (Bank, 2007; Byeong, 2006, de Jong, 2016; Gutek, 2014; Molina-Giron, 
2016; Von Bergen & Collier, 2013). Additionally, the Diversity Training Framework attributes emphasize social 
justice and inclusiveness (Gutek, 2014), based on tailoring approaches to meet individual needs (Hansen, 2007; 
Yirci, Karakose, & Kocabas, 2016), managing social interactions (de Jong, 2016), and developing competency in 
regards to differences present in the workplace as well as in society (Dimitrov et al., 2014). 
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Critical theory ethics are present within the Diversity Training Framework as well. Critical theory values 
such approaches as: dialogue, respect, support, equality, appropriate social interactions, individual beliefs and 
freedom (Gutek, 2014). Acceptance and civility, harmonic interactions and democratic citizenship, and valuing 
individual diversity are all framework strategies that align with critical theory ethical values (Bank, 2007; Byeong, 
2006, de Jong, 2016; Gutek, 2014; Molina-Giron, 2016; Von Bergen & Collier, 2013). Additionally, the framework 
tactics exhibit critical theory ethics as well. Communication, intercultural competency development, and coaching 
practices are directly aligned with dialogue, as they all foster open, respectful communication and knowledge and 
skill development (Dimitrov et al., 2014; Hansen, 2007; Gutek, 2014; Thakur & Thakur, 2008; Yirci, Karakose, & 
Kocabas, 2016). Migration and workplace management exemplify the critical theory ethics of freedom, respect, 
support, social interactions freedom through diversity encouragement and democratic workplace citizenship as well 
as equal, respectful treatment (Banks, 2007; de Jong, 2016; Gutek, 2014). Though critical theory may consider the 
globalized market aspect of the Diversity Training Framework to exploit people in other cultures (Gutek, 2014), the 
framework practices actively work to do the opposite, as the framework implements equality across geographic 
barriers (as well as other diversity barriers) to foster democratic citizenship and respectful, ethical equal treatment 
for all (Banks, 2007; Molina-Giron, 2016; Parker, 2003; Patrick & Kumar, 2012). 

Since critical theory does not meet the globalized needs of the Diversity Training Framework, considered 
ethics also expand into the neo-liberal ideology and the impacts of a globalized market (Gutek, 2014). Neo-liberal 
ideology focuses on globalization of, among other aspects of society, the economy and organization marketplaces 
(Gutek, 2014). With the modern version of organizations operating at international levels (Moravec, 2013; van den 
Hoff, 2014) diversity training practices, including those in the Diversity Training Framework, must consider 
diversity and organizations on a globalized scale and according to neo-liberal ideologies in order to embrace this 
globalization and meet the diverse needs of global citizens (Bresselink, 2013; De Leon, 2014; Gutek, 2014; 
Moravec, 2013; van den Hoff, 2014). Diversity Training Framework approaches, such as acceptance and civility, 
open communications across diversity obstacles and differences, democratic citizenship, and specialized training 
(e.g. coaching) to meet specific needs, address the concerns of a global organization and its respective diversity 
training needs (Banks, 2007; Dimitrov et al., 2014; Molina-Giron, 2016; Parker, 2003; Patrick & Kumar, 2012; 
Thakur & Thakur, 2008; Yirci, Karakose, & Kocabas, 2016). 


Conclusion 


It is not enough for change leaders to implement diversity training that employs the current approaches 
(e.g. equal employment opportunity, hiring in traditionally-marginalized population, training for behavior changes 
after negative interactions) to meet diversity needs within organizations, as this reactive approach alone does not 
ensure such ethical implications as equal, civil, accepting, effective organizations (Banks, 2007; Blum, 2007; 
Coates, 2014; Guerra, 2012; Gutek, 2014). Diversity training, and the change leaders that implement diversity 
training, must be both ethical and proactive and train personnel to value diversity, promote atmospheres of 
acceptance and civility in the workplace, provide a continued value of individual diversity and promote a collective 
effort to successfully implement diversity training (Byeong, 2006; de Jong, 2016; Guerra, 2012; Molina-Giron, 
2016; Von Bergen & Collier, 2013). Diversity training with ethical and effective strategies and tactics, such as those 
noted in the Diversity Training Framework, enable a diverse organizational environment that allows for continuous 
diversity evolution and ethical, positive workplace interactions (Byeong, 2006; de Jong, 2016; Dimitrov et al., 2014; 
Guerra, 2012; Molina-Giron, 2016; Thakur and Thakur, 2008; Yirci, Karakose, & Kocabas, 2016). 

An effective diversity training framework that can be applied within organizations is important to 
organization effectiveness and all personnel (Derderian-Aghajanian, 2010). Organizations should train personnel, 
while considering ethics, and create a culture of diversity understanding, which values diversity and diverse 
employee contributions, facilitates acceptance and respectful, civil behavior toward all employees and promotes 
cohesion and harmony among employees (Byeong, 2006; de Jong, 2016; Guerra, 2012; Molina-Girén, 2016; Von 
Bergen & Collier, 2013). A framework that meets these ethical and practical needs and enacts specific strategies and 
tactics, such as the Diversity Training Framework, can be applied within organizations to facilitate ethical and 
effective diversity training. This framework, and the strategies and tactics included within it, assist change leaders in 
creating an organization that values individuals as well as ethics while fostering a workplace that is effective, civil, 
ethical, moral, positive, accepting, and socially-just (Byeong, 2006; de Jong, 2016; Gutek, 2014; Molina-Giron, 
2016; Von Bergen & Collier, 2013). 
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Hey, want to Play? ‘Kahooting’ to Win the Learning Game 
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Introduction 


Using digital games in education or Digital Game Based Learning (DGBL) is becoming increasingly 
feasible due to the rise in variety and availability of digital educational and commercial games, as well as their 
consumption by contemporary users (Gros, 2007; Hainey, Connolly, Stansfield, & Boyle, 2011; Pivec, 2007). For 
example, in 2015, one hundred and fifty-five million Americans played video games, with at least two players in 
each game-playing household (ESA, 2015). Gaming is inherently motivating, which may allow learners to gain 
skills and knowledge, by leveraging entertainment and weaving it within learning environments (Becker, 2008; 
Bopp, 2006; Gee, 2005; Gee, 2003; Killi 2010; Killi, 2007; Miller, 2008; Paraskeva, Mysirlaki & Papagianni, 2010; 
Rieber, 1996; United States Department of Education, 2010; Van Eck & Hung, 2010). Research studies as well as 
conceptual literature indicate that video or digital games in particular have great engagement and education related 
benefits (Bogost, 2007; Griffiths, 2002; Paraskeva, Mysirlaki & Papagianni,2010; Zarraonandia, Diaz, Aedo & 
Ruiz, 2014). 

Currently, Digital Educational Games (DEGs) are emerging as interesting options for instruction, although 
they are not being used to their optimum potential in Higher Education (Godwin-James, 2014; Law & Sun, 2012). 
To institutionalize and adopt DGBL more extensively in Higher Education, it is important to research how using 
DEGS in Higher Education curriculum may affect learners’ experiences and learning (Epper, Derryberry, & Jackson, 
2012). Using a report published by National Foundation for Educational Research, Perrotta, Featherstone, Aston & 
Houghton (2013) indicate that although there is evidence that game-based learning can improve engagement and 
motivation, there is a gap pertaining to knowledge about its effects on performance outcomes. 

This mixed methods study seeks to contribute to literature by examining the value of using a DEG as 
compared to traditional teaching methods. The selected DEG, Kahoot, is gaining popularity in the educational arena, 
and according to Johan Brand (2016), CEO at Kahoot, 1.6 million Kahoots were played by 14 million players in 
over 100 countries in one month of January 2015 alone. However, at present there is no literature on the use of 
Kahoot. Thus, it would be useful to investigate this DEG in the context of improving student performance outcomes 
in Higher Education. 

Additionally, in order to understand why such improvements, if any, took place, the perceptions of learners 
with respect to the interest and challenge factors pertaining to the DGBL learning, must also be examined. Based on 
this, the study answered the following questions: 

1. Do learners have higher performance outcomes when exposed to activities involving Kahoot versus not 
using it? 
2. What factors within Kahoot do learners find interesting and challenging and why? 

The paper’s Background section provides overviews of key concepts pertinent to the study, including the 
theoretical frame. The Methodology section discusses elements of the study design, as well as the rationale for 
selecting Kahoot. Thereafter, Findings (quantitative and qualitative) are discussed. Finally, limitations and 
practitioners as well as research implications and conclusions are provided. 


Background 


DGBL refers to the inclusion of digital games as part of a curriculum and using them for different 
educational purposes like instruction and assessments (Nadolny & Halabi, 2015 Prensky, 2001; Van Eck, 2015). 
Within DGBL environments, learners usually participate in gameplay, explore various aspects of the game to satisfy 
the learning contexts created by the course designer or instructor, and engage in problem solving activities (Killi, 
2005; Nadolny & Halabi, 2015; Pivec, 2007; Pivec, Oziabenko & Schinnerl-Beikircher, 2014; Tsai & Fan, 2013). 

McClarty, Orr, Frey, Dolan, Vassileva, & McVay (2012) define DEGs as digital games that represent a 
technology-based system within which players engage in game generated competition that are guided by rules and 
have quantifiable outcomes. Such games seek to promote learning in an entertaining way by fusing educational 
content with gameplay and stories, while giving learners the opportunity to strategize with higher order thinking 
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(Kickmeier-Rust & Albert, 2012; Kickmeier-Rust, Gébel & Albert, 2008; Paraskeva, Mysirlaki & Papagianni, 2010; 
Pivec, Oziabenko & Schinnerl-Beikircher, 2014) Currently DEGS are being used in a variety of instructional 
settings, primarily in military and medical training (Alexander, Brunyé, Sidman, & Weil, 2005; Binsubaih, Maddock 
& Romano, 2009; Graafland, Schraagen & Schijven, 2012; Nicolaidou et al, 2015; Ulicsak, 2015; Wykes, 2012). 
Although many researchers support the use of DEGs (Bogost, 2007; Godwin-James, 2014; Griffiths, 2002; Law & 
Sun, 2012; Paraskeva, Mysirlaki & Papagianni,2010; Zarraonandia, Diaz, Aedo & Ruiz, 2014), others focus on the 
issues within DEGs, such as low entertainment value that makes such games less interesting for learners (Belotti, 
Ott, Arnab, Berta, deFreitas, Killi, & Oliver, 2011), and the dilution of effectiveness when education and 
entertainment are combined (Khine, 2011). Thus, when selecting any DEG, a judicious mix of entertainment and 
educational value must be considered, which is why Kahoot was the ideal choice for this study, as it is gaining in 
popularity, is easy to use, is free and has components of DEGs as described above. 

Additionally, after examining three DEG options (Kahoot, Quizziz and Socrative) that could be a fit for in- 
class activities, Kahoot was found to be the best choice, based on the criteria as shown in Table 1, and rationale 
provided by the participant instructors as adapted from Roediger (2015). 


Table 1. Comparing Kahoot, Quizizz and Socrative based on selection criteria for study use 


- Must have an account to administer and save games, preferably free,so Yes No Yes 
that instructors can share the same account and any edits and usage may 
be monitored. | i 
The Q and A should not appear on one screen. This was required to Yes No No 
avoid the ‘Blacberry Prayer” effect (Michaluk et al., 2010) where users 
are so glued to their device screens that they do not look up and /or 
_ participate in collaborative activities. | | | 
_ Ability to include images | Yes _| Yes | Yes j 
_ Ability to be used as a competitive game | Yes Yes ‘Yes 
_ Ability to be self-paced or timed _ Yes Yes Yes 
_ Teacher controlled as the purpose of use was to facilitate learning, and =_ Yes No Yes 
not assessment. Having teachers control the pace was essential. | | —_ 
Ability to search for games made by others that may act as extra Yes Yes No 
_Tesourcesifneeded a | Z 
_ Help and support is available _ Yes _ Yes _ Yes 


Kahoot is inherently entertaining and educational web based software for learning purposes that allows 
instructors and course designers to integrate instructional content through a quiz-like, gameplay construct. 
According to Singer (2016) it can be described as a television game show fused within a video game, where teachers 
act as game hosts and engage students. It provides players the opportunity for strategizing, problem solving, social 
interaction and collaborative learning within face-to-face class settings. The gameplay structure in Kahoot allows for 
quantifiable outcomes by mimicking quiz-based gameplay. To use Kahoot, teachers create learning games known as 
‘Kahoots’, using a series of multiple choice questions that can include videos, images and diagrams. Players answer 
on their own devices, while games are displayed on a shared screen and earn points for selecting the correct answer 
and potentially more points for how quickly they select the correct answer. The Kahoot games are point-based with 
winners and losers. Additionally, Kahoot can be used as an application on mobile devices. It provides an 
entertaining way to conduct in-class assessment, allowing teachers and students to evaluate student learning. 

Kahoot is gaining popularity within K-12 classrooms to enhance instruction in elementary math and junior 
high social studies and English (Knodel, 2016). According to Singer (2016), the company manufacturing Kahoot 
reported that 20 million out of 55 million elementary and secondary students in the United States used Kahoot in 
2016. Kahoot offers increased collaboration over conventional DEGs that rely on individual play modes (Stewart et 
al. 2013), and wherein. “subject mastery is generally emphasized over complex problem-solving” (Johnson, Smith, 
Levine & Haywood, 2010, p.18). As a quiz-based game, Kahoot can mesh with any discipline or subject matter as 
opposed to those DEGs that are designed for specific subjects (Johnson, Levine, Smith & Stone, 2010; Johnson, 
Smith, Levine & Haywood, 2010; Stewart et al, 2013). Despite its perceived value for education based on such 
reports, there are no studies on Kahoot’s value and viability in a Higher Education setting. 
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Methodology 
Mixed-Methods 


A mixed-method approach (Creswell, 2014; Creswell & Plano-Clark, 2007), using True Experimental 
methodology for quantitative and Phenomenology methodology for qualitative data, was used to examine learner 
performances and experiences within a DGBL environment. Mixed-methods have been known to provide deeper, 
more comprehensive data and result in superior research (Halkier, 2011; Johnson & Onwuegbuzie, 2004; 
Schreder, 2012). For this study, following Creswell’s (2014) convergent-parallel mixed —method approach, 
collection of quantitative and qualitative data was done at approximately the same time and then analyzed by 
converging information from the data. The three key identifiers of the True Experimental design are randomization, 
control and manipulation (Johnson & Christensen, 2014; Berry & Yost, 2013), which the study incorporated. The 
class sections were selected randomly and the duration of usage of Kahoot was controlled within the experimental 
groups. Participants were randomly placed in experimental (using Kahoot) and control (not using Kahoot). 
Additionally, the study used a Phenomenological approach for Qualitative data that was drawn from participant 
interactions and reactions to a DGBL based learning environment, as suggested by several researchers of this 
methodology (Creswell,2014; Holroyd, 2001; Husserl, 1931; Sadala & Adorno, 2001). 


Setting and Participants 


Participants were selected from one Midwestern Community College Campus with an approximate 
population of 5,100 The sample population were 96 students from Introduction to Business courses, enrolled in 
Associates Degree programs, recruited from five sections taught by three instructors who agreed to participate. 
Based on information from the course instructors and Introduction Discussion forum posts, participants were 
approximately 18 to 60 years of age and were mixed in terms of ethnicity. Based on survey data, 90% of the 
population had little to no experience with Kahoot. All of the instructors had no prior experience with DGBL or 
gaming in general; although they were knowledgeable in computer usage and basic educational technology. 


Data Gathering 


Course identifying numbers for the participating instructors were randomized to select sections for the 
experiment. Quantitative data was collected using the scores participants received for their Final exams. Within the 
experimental sections, students were taught course materials using lectures that were complemented by Kahoot 
based activities, versus in the control group where instruction was limited to lectures. The control group courses did 
not use Kahoot at all. 

Instructors created ‘Kahoots’ or interactive and problem-solving based question and answer activities, 
using material from their lectures and the common text books assigned to the course. In the classrooms, these 
Kahoots were used as different group activities. These were usually done after a lecture. For example, some days, 
students played a Class-Jeopardy game. On other days, they played a group contest game based on which group had 
more correct answers or which group was more prompt in responding to the questions. At least once during the 
experiment, students played a round table game wherein groups designed their own questions based on the ideas 
generated by the Kahoot, and quizzed other groups. The focus of these gameplays was on interactive and 
collaborative learning. For each activity, groups played against other groups for points. 

The final exam scores were compared to examine the effects of the Kahoot intervention. The exam 
questions for both groups were same. The final exam was held at the end of the semester, and had 100 multiple 
choice questions, worth one point each. This exam covered all materials for the entire semester for the course. 
Students had two hours to complete this exam. Students were not allowed to use any notes or consult the text book 
for this exam. 

Qualitative data was collected using one survey with three questions, and entries made in one reflection 
journal. The survey questions were: “Have you experienced Kahoot type of learning technology before? If yes, what 
technology was it? If not, say NA. Write down three words or sentences that identify your feelings about using 
Kahoot? Would you like to engage in Kahoot based activities again? Why or why not? Explain clearly, but briefly”. 


Analysis 


For the exams. Independent sample t-Tests were conducted, and Cohens d effect size was calculated. 
Survey data was ‘quantitized’ (Johnson & Christensen, 2014, p, 504) using term and phrase frequencies showing a 
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comparative analysis of the perceptions of Kahoot as expressed in specific words/phrases. First, a list of all 
words/phrases used by participants was drawn. Since several of these were overlapping and more than one 
participant used them, the number of responses per word/phrase was counted, then codified based on the frequency, 
and represented using a clustered bar graph. 


Results 
Research Question # 1: Performance Outcomes 


The HO was that there will be no significant difference between the exam scores of learners using Kahoot 
and students from control groups. The HA was that learners using Kahoot will have significantly higher exam 
scores than control groups. Due to directional approach for the HA, one-tail, independent sample t- Tests was used. 


Alpha-level for both tests were set at 5% ( a _ .05). Confidence level was set at 95%. Based on the results of the 
t-Test and Cohen’s d, the null hypothesis (HO) was rejected. Table | highlights the results of the t-Test using 
numbers and graph. 


Table 1. Descriptive Statistics and T-Test Results 


P-value is <] Significant 
T-Value is 6.90809 .00001 at p< .05 


PN Minimum | Maximum | Mean_| Std. Deviation 


Experimental 48 43.00 100.00 79.5625 13.10174 
Control 48 25.00 92.00 56.8333 18.65400 
Valid N (listwise) 48 


100.00-) 


80.00-] 3 


Control 


60.00 . 


40.007 = 


20.004 


Experimental 


The effect size using Cohen’s d was large: Cohen's d = (56.83 - 79.56)/16.116121 = 1.410389. 


Research Question # 2: Interesting and Challenging Factors 

Data revealed that none of the participants had played Kahoot in their classrooms before; although, some 
had minimal experience with other in-class gaming. This supported the researchers’ findings from literature that 
Kahoot has been underutilized in Higher Education, despite its perceived value for educational purposes as found in 
K-12 environments. All participants found the use of Kahoot to be extremely beneficial, and 100 percent of them 
agreed that they would like to engage in Kahoot based activities again. When answering questions about using 
words/phrases to identify their feelings about Kahoot and why they would like to use it again, participants came up 
with 35 words/phrases in total, many of which were repeated by different participants, bringing the total number of 
words/phrases submitted to 103. Based on the codification, these were divided into seven categories that identified 
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the general perceptions about the interesting and challenging aspects of Kahoot. Figure | provides a cluster graph of 
the results. As revealed in Figure 1, participants found Kahoot to be highly entertaining, motivating, exciting and 
helpful in providing assistance to their learning. 


Perceptions of Kahoot Value 


Motivating 
Learning Assistance 
Helpful 

Exciting 
Entertaining 
Diitatlie 


Competitive 


Figure I Number of words/phrases submitted per category. 


Most participants did not find using Kahoot difficult or challenging. Finally, only some participants found 
Kahoot to be competitive, even though it was used to create competition among groups in class. In this study, 
participants appear to have been more involved due to the entertainment factors and the perceived value they found 
in using DEG based activities for their learning purposes. This was further collaborated by participants’ journals, 
excerpts from which are given below using pseudonyms. 

One of the values seen was the ability of such activities to help review course materials and help them 
prepare in a better way for the exams. For instance, Emma stated, “I like the Kahoot technology, I think it is a better 
way to review the material, and get ready for the tests”. Kyle noted “It was fun and seemed to have helped refresh 
the reading”. Some participants felt that this helped them retain information better, thus assisting their working 
memory. For instance, Sherry explained how “Kahoot was very helpful it just refreshed what I learned and put it in 
my long-term memory”. In a similar vein Darren mentioned, “About the kahoot i like that it reinforced the stuff we 
were learning and made it more likely to be remembered”. Additionally, participants’ ability to read and take notes 
were also facilitated. Charlene confided how, “Before using Kahoot I mostly read the power points and took short 
notes, but after we started playing I would take better notes and read over them to make sure I understood the 
definitions”. Finally, using Kahoot facilitated more robust participation, even by shy students. For instance, Sam 
mentioned that, “I liked playing Kahoot in class. It involved people in class that are scared to speak out to tell the 
right answer when you ask questions”, while Emma stated, “It is also a good way to get people to participate in the 
class”. 


Integration of Qualitative and Quantitative Findings 


A mixed method approach complemented quantitative data with qualitative findings to gain a 
comprehensive understanding of the phenomenon of using Kahoot based intervention in classrooms. While the 
quantitative data indicated that there were definite improvements in performance outcomes for the experimental 
group versus the control group, the qualitative analysis provided several pointer as to why this could be. The key 
factor in this was the element of motivation that the Kahoot usage provided that could be deemed as the catalyst for 
the improved performances. Despite some issues and concerns regarding technology based instruction (Ariffin, 
2012; Ertmer, 2005), technology integration is seen as a criterion for pedagogical expertise and superior 
instructional design (Borthwick et al, 2008; Jaipal-Jamani & Figg, 2014; Piersen, 2001). 

Table 2 provides a synthesis of the key qualitative findings and how they relate to the quantitative findings 
of higher performance, with supporting literature. When examining the qualitative data from this study the following 
factors were offered by participants as crucial to their engagement with the games, that perhaps steered their 
motivation to higher performance. 
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Table 2. Quantitative and Qualitative Findings Synthesis 


Interesting Kahoot contents, including audio-visual Repeated readings can lead to higher performance 
components, that helped students repeatedly read (Chafouleas, Martens, Dobson, Weinstein, & Gardner, 
content and the associated engagement factors led to 2004; Granic, Lobel, Engels, Anderson, N. B., 2014; 
repeated interaction with such contents. Schlickum, Hedman, Enochsson, Kjellin, & 


Fellander-Tsai, 2009; Therrien, 2004). 


Socializing options within video games can enhance _ Socializing based motivation has a higher compelling 
motivation. impact on learners’ efforts and performances 
(Boggiano & Pittman, 2010; Lee, Hwa Hsu & Chang, 
2013; Peterson, 2013). Having a competitive essence in 
an activity may enhance engagement and allow 
learners to develop more effective problem-solving 
skills and participate more robustly in uninteresting or 
mundane educational activities, which may lead to 
higher learning (Annetta, Minogue, Holmes, & 

Cheng, 2009; Burguillo, 2010). 


Discussion 


As is evident from the results, the use of Kahoot positively affected learners’ performances. The effects 
were seen with respect to higher performance in exams and greater cognitive retention, a fact that several 
participants discussed in their reflection on the game. For the final exam, contents from fourteen weeks were used. 
Thus, for the final exam, students had substantial cognitive load when compared to the rest of the semester, where 
assignments covered smaller chunks of subject matter. However, for seven weeks preceding the final exam, content 
was taught using Kahoot. Additionally, since it is possible that after a gap of several weeks, students would have 
forgotten some of what they learned during the first half of the semester (which did not use Kahoot), in order to 
perform well, students may have had to put in more effort for the final exam. It is possible that this effort was 
boosted by the use of Kahoot. 

Cowan (2010) discussed storage- specific measuring in the context of the nature of working memory 
limitations, and posited that forgetting some of the acquired learning may lead to shallower understandings, and 
learners would thus need to go back and reread materials. He also stated that the limits of working memory ability 
differ, based on how much a person can store and how effectively working memory is used. For instance, “An 
important example is in the use of attention to fill working memory with the items one should be remembering (say, 
the concepts being explained in a class) as opposed to filling it with distractions (say, what one is planning to do 
after class)” (Cowan, 2010, p. 52). With respect to the participants for this study, the time gap between learning 
activities would be enough to create several distractions, thus reducing their working memory capacity. Yet, the 
experimental students’ performance in the overall final exam was better than the control groups’. 

Qualitative results strongly indicated that participants were engaged by the use of Kahoot in the classroom. 
This supports evidence in the literature that students find digital game related activities more entertaining and 
motivating, and that these factors in turn may assist learning (Barab, Thomas, Dodge, Carteaux, & Tuzun, 2005; 
Bogost, 2007; Griffiths, 2002; Hainey, Connolly, Stansfield & Boyle,2011; Paraskeva, Mysirlaki & 
Papagianni,2010; Zarraonandia, Diaz, Aedo & Ruiz, 2014). One unexpected result was that despite the game-show 
aspect of Kahoot and its focus on competition between teams, participants did not identify the game as a competitive 
experience. This is different than what is talked about in some studies and conceptual literature that emphasizes the 
competitive value of games as being key to participants’ interest and motivation (Annetta, Minogue, Holmes, & 
Cheng, 2009; Burguillo, 2010; Jayakanthan,2002; Ladd, & Fiske, 2003; Papastergiou, 2009; Vorderer, Hartmann & 
Klimmt, 2003). Despite the lack of competition, students clearly were engaged, and not only by the gaming 
experience, but also their perception of the value of the game for learning. 
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Implications, Limitations, Conclusion 


In terms of research implications, there are strong indications that the DGBL based intervention using 
Kahoot was successful in delivering higher performance scores and inciting high interest and engagement in 
participants as opposed to traditional teaching methods. A limitation of the study is that it only examined the use of 
one DEG, However, when viewed through the lens of this study’s findings, it can be extrapolated that DGBL 
strategies using DEGs may positively affect performances. Investigation of other DEGs in this context may be 
beneficial. Based on the success of the study and suggested value of Kahoot as a DEG for enhancing performance 
outcomes, it is recommended that the study be continued to replicate the experiments. Additional limitations are that 
this study focused on learners, and did not include the teachers’ perceptions, which may be crucial to get a deeper 
view of the value of this Kahoot based intervention. Furthermore, it will be useful to examine the effects of Kahoot 
DEG usage in different time slots. For instance, it will be useful to see learner reactions when Kahoot is used only 
for the first half of the semester in some courses and vice versa in others. Finally, it will also be valuable to examine 
the use of Kahoot in a variety of subject areas and as assessment tools, since this study’s focus was on using it as a 
knowledge regulator. 

In terms of practitioners’ implications, instructors should look to DGBL interventions as tools that can 
potentially increase engagement to a point where performance is positively impacted. To conclude, the ability of 
Kahoot to help learners with motivation and performance is highly encouraging and significant. Kahoot provides an 
example of a free game with limited barriers to use in traditional classroom environments. It and games like it are, 
therefore, worthy of further examination as they hold strong potential for accelerating the incorporation of DGBL in 
the classroom. 
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Abstract 


A 2x 2 factorial design experimental study was conducted to investigate the effects of prior knowledge and 
visual cues in a complex animation on learning and metal effort. It also examined whether the effect of visual cueing 
is moderated by the prior knowledge. Data were collected from 102 undergraduate students from various majors in 
the Southwest University. The results showed that effects of cueing in a complex animation change depending on 
learners’ level of prior knowledge. Specifically, low prior knowledge learners benefited more when visual cues were 
provided, whereas cues did not facilitate learning for high prior knowledge learners. 


Theoretical Background 


Multimedia learning environments have the ability to present the information in textual, graphical, and 
audible forms (Mayer, 2001). This allowed the instructional designers to create more effective and richer learning 
environments. However, the design of multimedia instruction is crucial in order to provide effective instruction to 
the learners. According to multimedia principle, people learn better when the information is presented with text and 
pictures than with words alone (Mayer, 2005). In some situations, using only words with pictures is not necessary to 
enhance the learning. According to Cognitive Load Theory (CLT) and Cognitive Theory of Multimedia Learning 
(CTML), the human mind has limited capacity of processing the information in the working memory at the same 
time (Mayer, 2001; Sweller, 2005; Sweller, van Merrienboer & Pass, 1998). Therefore, only presenting the 
information with multiple modalities is not enough to ensure the superior performance when we consider the 
limitations of the human cognitive system (Ginns, 2005; Sweller et al., 1998). As an initiative to ensure the superior 
performance in multimedia learning environments, recent research studies have begun to focus on the use of 
strategies to reduce extraneous cognitive load and enhance the intrinsic cognitive load. 

With the advances in the computer technology and software, instructional designers have been widely used 
to create more effective and interactive learning environments to help the learners enhance their recall of complex 
dynamic systems and their understanding of the materials (Lowe, 2004). Animations provide several advantages, 
such as increasing interactivity of instructional materials, enhancing engagement, object trajectory, depicting motion 
and making abstract information concrete (Betrancourt, 2005). However, including animations in multimedia 
learning environments is not enough by itself to ensure superior performance unless we consider the limitations of 
the human cognitive system. 

Figure | represents the human information-processing system according to the CTML. It is represented as a 
series of boxes and arrows. The big rectangular boxes indicate the memory stores that are sensory memory, working 
memory and long term memory. Two rows represent the dual-channels, with the auditory/verbal channel in the first 
row and visual/pictorial channel at the bottom. The arrows represent the cognitive processes. The arrow from word 
to eye indicates the action of the printed text entering through the eyes. 


67 


MULTIMEDIA SENSORY LONG-TERM 
PRESENTATION MEMORY WORKING MEMORY MEMORY 


Figure 1. Cognitive theory of multimedia learning. Adapted from Multimedia learning (p.61), by R.E. Mayer, 
2009,New York: Cambridge University Press. Copyright 2009 by the Cambridge University Press. Adapted with 
permission. 


Prior 


Knowledge 


CTML defines five cognitive processes which are required for meaningful learning to occur animation with 
narration. These are selecting relevant words, selecting relevant images, organizing selected words, organizing 
selected images and integrating word-based and image-based representations (Mayer, 2009). Due to the limited 
capacity of the working memory, it is essential to select relevant words and images to process. The learner is 
responsible for judging the images and words and choosing the most relevant one in order to understand the 
presented information. However, learners might difficulty in selecting the essential information from the visual 
display which corresponds to the narration within a limited time due to the transience nature of the animation. 
Learners are required to keep the narrated information in the Working Memory (WM) while searching to the 
relevant information from the visual. This causes a cognitive load in the WM. Moreover, learner might miss the 
information, if they cannot find the relevant information within a limited time. Therefore, educational researchers 
have paid considerable amount of attention to learning from animations (Ploetzner & Lowe, 2004). 

To make animations effective for learning, the cognitive load caused by the transience should be 
counteracted. Several researchers (Boucheix, Lowe, Putri & Groff, 2013; Sheiter & Eitel, 2015) have stressed the 
importance of guiding learners’ attention to the relevant information in animation. This can be done by the use of 
cues. Cueing guides the learner’s cognitive processing, helps the learner to select the relevant information, organize 
the information logically, and integrates the information with prior knowledge by providing cues (Mautone & 
Mayer, 2001). 

Besides cueing, individual difference needs to be considered in instructional design (Betrancourt, 2005; 
Kalyuga, 2009; Kalyuga, 2014; Mayer, 2001). Henderson (2003) stressed that prior knowledge in the long-term 
memory (LTM) can control the learners; visual attention. “The selection of essential visual information of a display 
depends not only on the manner in which the information is presented, but also on the quantity and specificity of the 
pre-existing knowledge” (Khacharem, 2016, p.2). Prior knowledge plays an important role on distinguishing the 
relevant and irrelevant information, so it helps the learners to direct their attention to the essential information 
(Jarodzka, Scheiter, Gerjets, & van Gog, 2010). Expertise reversal effect also stressed that design principles that are 
effective for low-knowledge learners may not help or hinder high-knowledge learners (Kalyuga, 2014). 

Guidance effect of cueing have been highlighted by several research studies (e.g., Ozcelik, Arslan-Ari, & 
Cagiltay, 2010; Jamet, 2014). However, there are limited number of studies examining the interaction between prior 
knowledge and cueing, which is one of the limitations of cueing research in multimedia learning (Richter, Scheiter 
& Eitel, 2016). Though few studies (Johnson, Ozogul, & Reisslein, 2015; Khacharem, 2016) found a significant 
interaction effect when cues were provided with the features of animated pedagogical agents and arrows in a static 
diagram, very little is known whether visual cueing in a narrated animation interact with prior knowledge to affect 
learning. Thus, the purpose of this study was to examine the effects of prior knowledge and visual cueing in a 
complex animation on learning and mental effort. It also investigated whether the effect of visual cueing is 
moderated by the prior knowledge. 
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Methodology 


A total of 102 volunteer undergraduate students from various majors in a large Southwestern university 
participated in the study. To categorize the participants as low prior knowledge (LPK) and high prior knowledge 
(HPK), 33th and 66th percentiles of their prior knowledge scores were used as cut-offs. The students whose prior 
knowledge scores were in the 33th percentile or below were assigned to a LPK group (n=36) whereas the students 
whose prior knowledge scores were in the 66th percentile or above were assigned to a HPK group (n=37). 
Therefore, only 73 participants (39 females, 34 males) were included in the data analysis. Their ages ranged from 18 
to 34 with a mean of 20.34 and standard deviation of 2.81. An independent t-test revealed that the prior knowledge 
of two groups were significantly different, ¢ (71) = - 29.32, p<.01. 

Participants were randomly assigned to either no cueing (n =36) or visual cueing (n =37) conditions. This 
resulted in four experimental groups: no cueing/LPK (n=19), no cueing/HPK (n=19), visual cueing/LPK (n=17), and 
visual cueing/HPK (n=20). 

Two versions of an instructional material about photosynthesis, light dependent and light independent 
reactions were used. The content was adapted from a college level plant biology textbook presented with four 
separate animation. The cued animation was identical to the uncued one except the presence of the visual cues (see 
Figure 2). The animations took totally 9 minutes and 47 seconds. In cued animation, each corresponding 
terminological label (e.g. Photosystem I, P700, plasocyanin) became red during the narration of the sentence in 
which the item was mentioned. After each animation, two buttons, “Continue” and “Replay”, appeared. The learners 
were not allowed to go backward or to skip around within the program. 
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Figure 2. Sample screenshot of no cueing (/eft) and visual cueing (right) conditions/ 


This experimental study was conducted in a computer laboratory (15-24 participants per session). After the 
participants were given introductory information about the study, they were asked to complete a demographic 
questionnaire and prior knowledge test. Then, they studied the instructional material at their own pace and 
completed the computer-based tests consisting of the mental load scale, retention test, the transfer test, and the 
matching test sequentially. There was no time limit to study the material and to complete the tests. 


Results & Discussion 


A two-way MANOVA was conducted to determine the effects of prior knowledge and cueing on retention, 
transfer and matching. MANOVA revealed a nonsignificant main effect for cueing (Wilks’s = .99, F(3, 67) =.23, 
p=.88) and a significant main effect for prior knowledge (Wilks’s = .880, F(3,67) = 5.75, p< .05, ?=.21). 
However, the interaction between prior knowledge and cueing was also significant (Wilks’s = .84, F(3, 67) = 4.14, 
p<.05, *=.17). Given multivariate significance, the univariate effects were examined. Univariate testing indicated 
this interaction to be significant for transfer (F' (1,69) =6.93, p<.05, 7? = .06) and matching (F (1,67) =5.38, p<.05, 7? 
= .07. 

Follow-up pairwise comparisons indicated that LPK learners in the visual cueing group had significantly 
higher transfer scores than the LPK students in the no cueing group, p< .05. Although there was no significant 
difference in matching scores, the examination of the mean scores indicated that LPK learners in the visual cueing 
(M=55.66) outperformed those in the no cueing group (M=44.42) on the matching test. 
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For HPK learners, matching (p=.35) and transfer scores (p=.17) did not significantly differ between no 
cueing and visual cueing groups. However, HPK learners did better on the transfer and matching test when the 
animation does not include cues. 

When there was no cueing, HPK learners had significantly higher matching and transfer scores compared 
to LPK learners, p<.05. However, when visual cues were introduced, although it is not statistically significant, LPK 
learners had higher transfer and matching scores than HPK learners. 

The follow-up univariate tests’ results revealed a nonsignificant interaction effect on retention 
F(1,69) = .07, p =.79. 

The results of the two-way ANOVA revealed that there were no significant effects of prior knowledge 
(F(1,69) = .94, p = 34), cueing (F(1,69) = .17, p = .69), or interaction effect (F(1,69) = .22, p = .64) on mental 
effort. 

These results indicated the moderating effects of prior knowledge on the impact of cueing in learning from 
instructional animations. In this study, LPK learners benefited more from cueing. The findings are consistent with 
the expertise reversal principle that emphasizes instructional design principles for multimedia learning which 
enhance LPK learners’ performance may not be equally effective for HPK learners (Kalyuga, 2014). These design 
strategies may even hinder the performance of high prior knowledge learners. Providing external visual guidance in 
an animation might impose an unnecessary load on the WM of high prior knowledge learners and diminish 
resources necessary to gain and integrate new information because they still need to relate the existing knowledge 
base with externally provided guidance (Kalyuga, 2008). 
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Abstract 


The purpose of this study is to assess the adaptation of the newly developed Computer Science (CS) 
curriculum in middle schools and address teachers’ challenges during its implementation. Since CS has been 
introduced as a new subject in K-12 schools in the U.S., it is essential to determine the current implementation of CS 
curriculum from teachers’ perspectives in order to enhance the CS education movement. Considering the fact that 
curriculum implementation interacts with multiple factors in K-12 school systems, this study assesses the current 
status of CS curriculum implementation in middle schools based on four categories: (1) teacher, (2) curriculum, (3) 
context, (4) students. The participants were middle school teachers who taught Introduction to Programming course 
in the Southwestern United States. The findings showed that teachers had positive attitudes toward CS education and 
received enough opportunities for professional development in CS. However, teacher sought effective instructional 
strategies to enhance students’ understanding of CS concepts and better engage their students in CS learning. As for 
the curriculum, teachers perceived the scope, sequence and learning objective of the CS curriculum were appropriate 
for the purpose, whereas they needed more time to enhance students’ understanding of CS concepts. In addition, the 
findings suggested that some schools do not have appropriate equipment or facilities. Based on our findings, 
recommendations for implementing CS curriculum to support CS education and implications for future research are 
presented and discussed. 


Introduction 


Computer Science (CS) is becoming an essential component of students’ educational development for 
computational thinking skills including problem-solving skills, analytic thinking skills, and creativity (Atmatzidou 
& Demetriadis, 2016; Papasterigiou, 2009). In an effort to promote these skills, there are numerous CS education 
initiatives. Nationally, Computer Science for All called for a $4.1 billion budget to support K-12 teacher 
professional development and provide high-quality instructional resources (Office of the Press Secretary, 2016). 
Further initiatives including Code.org, CS first, and Code Academy have expanded access to instructional materials 
to support K-12 CS education. 

In recent years, new CS curricula have been developed and implemented in K-12 schools in the U.S., and 
school policy has been changed to categorize CS as a part of the math and science core curriculum across states 
(Alba & Huett, 2017; Nager & Atkinson, 2016). For example, at the high school level, CS counts as a core 
graduation requirement in 23 states and the District of Columbia (Stanton et al., 2017). Specifically, Arkansas, 
Texas, Virginia, and West Virginia now require public high schools to offer at least one CS course. At the middle 
school level, CS education has also grown. For example, several states including Nevada and Indiana, aim to 
promote student access to CS education through computing courses in elementary and middle schools (Stanton et 
al., 2017). Despite some efforts and initiatives to enhance middle school CS education, less attention has been given 
to middle schools compared to the high school level. Thus, to promote CS education in middle schools, CS 
curriculum for middle schools’ students needs to be developed and further supported. 

In addition to school policy, the current adaptation and implementation of new CS curriculum should be 
evaluated to better support the new movement. Several researchers have noted that teachers may face challenges 
throughout the new curriculum implementation process (Brown et al., 2013; Finger & Houguet, 2009; Sentance & 
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Csizmadia, 2017; von Wangenheim, Alves, Rodrigues, & Hauck, 2017). For example, it requires them to changes in 
both subject matter and pedagogical knowledge (Alba & Huett, 2017; Thompson et al., 2013). Since this may be an 
unfamiliar area for some teachers, they may not have enough experience to teach CS contents, or they may feel ill- 
equipped to teach new contents knowledge. As a result, they may need to seek appropriate pedagogies for teaching 
CS, such as algorithms, or programming, to facilitate students’ computational thinking skills. However, we have yet 
to understand how teachers have adopted the newly-developed CS curriculum and how they perceive their 
adaptation. In order to understand the current status of the CS curriculum and identify areas of improvement, the 
challenges that teachers encounter must be examined. 

Furthermore, since school systems interacts with multiple variables when implementing a new curriculum 
in classrooms, we need to highlight other relevant factors, such as students and context, which may affect the new 
CS curriculum in K-12 school settings. For example, possible issues would be lack of students’ readiness for 
learning CS, lack of students’ engagement, appropriateness of scope and sequence of developed curriculum, and 
lack of infrastructure across schools (Sentance & Csizmadia, 2017; Wangenheim et al, 2017). Considering that 
teachers, students, and school contexts vary and interact together in dynamic ways when implementing a new 
curriculum across schools (Remillard, 2005), it is critically important to explore multiple factors influencing the CS 
curriculum implementation so that we can provide a holistic diagnosis of the current status of CS education 
initiatives. 


Purpose of the Study 


The purpose of this study is to assess the current adaptation of newly developed CS curriculum and identify 
teachers’ challenges and the research questions of this study are: 

1. How have teachers adopted the new CS curriculum? 

2. How do teachers perceive their adaptation to the new CS curriculum? 

3. What are the challenges to implementing the new CS curriculum? 


Method 
Context and Participants 


The “Introduction into Programming” was developed as a CS education initiative by the College of 
Education with the support of an urban school district funding in 2016. It was implemented in the 30 public middle 
schools within the school district. Participants in this study were 12 middle school teachers (25% males, 75% 
females) who taught the course using the newly developed curriculum in 2016. 


The CS Curriculum Design 
The “Introduction into Programming” curriculum includes various activities, such as storytelling, 
animation, and games using Scratch, Google CS First, and Code.org. This eighteen-week long curriculum consists 


of 75 lessons, and the first and second six weeks focus on different CS concepts with unplugged activities and 
programming projects, and the third six weeks focus on HTML and CSS (See Figure 1). 
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Figure 1. Students CS Website 


Data Collection and Analysis 


Online-Survey. By referencing the components of the teacher-curriculum relationship proposed by 
Remillard (2005), we developed four constructs to assess the curriculum implementation. Our proposed framework 
includes these four categories: (1) teacher, (2) students, (3) curriculum, (4) context. The survey items adapted from 
previous studies (Friday Institute for Educational Innovation, 2012; Sharp, Hopkin & Lewthwaite, 2011; Ko & Lee, 
2003; Ross, McDougall, Hogaboam-Gray & Lesage, 2003) which were modified to fit our research context in order 
to measure the CS curriculum implementation. The online survey consisted of 83 Likert-scale items with 15 open- 
ended questions. The survey included questions about how teachers used the curriculum in their teaching and how 
they perceived the CS curriculum implementation using the four categories. The open-ended questions explored 
teachers’ challenges in implementing the CS curriculum. 

Data Analysis. Descriptive statistical analyses were performed on the survey data to investigate the 
implementation of the new CS curriculum and teachers’ perceptions to the new curriculum. Content analysis was 
used to examine data from open-ended responses in the survey to identify teachers’ challenges in teaching CS. 


Procedure 
The online survey was distributed to thirty middle schools by means of the school district office in spring 


2017. Among 30 middle school teachers, 12 teachers completed the survey and their responses were included in the 
analysis. 
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Results 
Use of the New CS Curriculum in Teaching 
To answer the first research question, we investigated how the CS teachers used the new curriculum in their 
instruction. As shown Table 1, out of the 8 modules, teachers used the first five modules, but only two teachers used 


the module to introduce HTML, and no one used the last two modules. 


Table I. Used modules in CS course (N=12, Multiple responses) 


Modules Used Module (N) 
First Six Weeks 
M1- Welcome to programming 11 
M2 — Tell your story 10 
M3 — Be an artist 10 
Second Six Weeks 
M1 — Game Design 12 
M2 - Animation 9 
Third Six Weeks 
M1 - Getting Started with HTML 2 
M2 - Getting Organized with HTML 0 
M3 — Getting Stylish with CSS 0 


In addition, teachers were asked which module was the most interesting and challenging for their students. 
An analysis of survey data showed that 75% teachers responded that the “Game design” module was the most 
interesting, and 25% of teachers replied that “Game design” and “Animation,” respectively, were the most 
challenging modules for their students (See Table 2). 


Table 2. The most interesting and challenging modules for students (V=12) 


ER HIES The most interesting The most challenging 
module module 
First Six Weeks 
M1- Welcome to programming 2 (16.67%) 1 (8.33%) 
M2 — Tell your story 1 (8.33%) 0 (0.0%) 
M3 — Be an artist 0 (0.0%) 1 (8.33%) 
Second Six Weeks 
M1 — Game Design 9 (75.0%) 3 (25%) 
M2 - Animation 0 (0.0%) 3 (25%) 
Third Six Weeks 
M1 — Getting Started with HTML 0 (0.0%) 2 (16.7%) 
M2 - Getting Organized with HTML 0 (0.0%) 1 (8.33%) 
M3 — Getting Stylish with CSS 0 (0.0%) 1 (8.33%) 


Teachers Perceptions of Their Adaptation to the New CS Curriculum 


The second question was about how teachers perceived their adoption of the new CS curriculum. The four 
categories (i.e., teacher, curriculum, student, context) were used to evaluate the CS curriculum implementation. In 
the teacher category, our analysis of survey data showed that teachers had positive attitudes toward CS education (M@ 
= 4.83, SD = .39). For example, 83% responses strongly agree with the statement: “I support the inclusion of 
Computer Science education in middle school curriculum” (M = 4.83, SD = .39). Moreover, all teacher responses 
either agree or strongly agree with the statement, “I am confident that I can explain key Computer Science concepts 
to students with examples” (M = 4.5, SD = .52), suggesting that teachers showed high confidence in teaching CS 
(See Table 3). 
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Table 3. Results of teachers’ perceptions of their adaptation: Teacher (N=12) 


Construct Number of Items Cronbach’s o M SD 
Attitude toward CS education 3 1.00 4.83 39 
Subject Knowledge b) 53 4.37 .40 
Teaching Efficacy 3 .96 4.56 50 
Intention 3 .76 4.03 87 
Total 14 711 4.43 35 


Regarding the curriculum, the results showed that teachers overall perceived the scope, sequence and 
learning objective were appropriate for the purpose (See Table 4). Among the constructs, the mean scores of 
materials was high (/ = 4.06, SD = .66), indicating that teachers perceived the student curriculum website was 
useful for students and their teaching. On the other hand, instruction time was the lowest score (M = 2.83, SD = 
1.10). For example, in response to the survey item, “Each lesson’s learning activities could be completed within the 
given instructional time,” over 42% of the responses either strongly disagreed or disagreed. 


Table 4. Results of teachers’ perceptions of their adaptation: Curriculum (N=12) 


Construct Number of Items — Cronbach’s a M SD 
Scope 3 97 3.86 86 
Sequence 3 93 3.69 82 
Learning Objectives 3 98 3.67 99 
Standards 3 1.00 3.83 83 
Complexity 3 96 3.58 95 
Time 3 98 2.83 1.10 
Materials 3 899 4.06 66 
Promoting Higher-Order Thinking 3 82 3.75 74 
Assessment 3 95 3.19 99 
Total 27 96 3.61 64 


In contrast to the teacher side, the mean of teacher perceptions of students was the lowest in the four 
categories (M = 3.33, SD = .29). An analysis of survey data indicated that the students lacked prior programming 
skills (M = 2.33, SD = .72). For example, over 66% of teachers either strongly disagree or disagree with the 
statement: “My students have prior programming experiences” (M = 2.08, SD = .79). To gain in-depth 
understanding of students’ challenges, we asked teachers what challenges their students faced in the CS course. An 
analysis of open-ended questions suggested that students’ lack of prior programming skills may impact individual 
students understanding of CS concepts and instructional time. In open-ended responses, many teachers mentioned 
that their students have no prior programing experience and have a hard time understanding the idea of essential 
concepts. For example, one teachers commented, “They are unable to use the concepts introduced by the CS First 
activities to create original programs on their own” [Teacher 3]. 


Table 5. Results of teachers’ perceptions of their adaptation: Student (V=12) 


Construct Number of Items — Cronbach’s a M SD 
Prior Computer Skills 3 93 4.11 78 
Prior Programming Skills 3 .64 2.33 72 
Interests 3 .96 3.67 71 
Difficulty” 3 .80 3.22 87 
Total 12 71 3.33 29 


* Reversed items 


Results also revealed that teachers needed more time to enhance students’ understanding of CS concepts. 
As some teachers stated, teachers spent instructional time in explaining basic information rather than engaging them 
in programming activities. As a result, teachers felt they did not have enough time to complete the designed 
activities. As one teacher mentioned, “Majority of students minus a handful of 130 students have no experience in 
any capacity with regards to programming, coding, coding concepts, or all of the above. I had to baby step them in 
more slowly than the curriculum called for but in the end it worked out well......”. [Teacher 4]. 
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As for the context, overall, the teachers perceived that they received enough opportunities for professional 
development in CS (VM = 4.11, SD = .73), and have received appropriate administrators support (/ = 3.89, SD =.76) 
and technological support (M = 3.47, SD = .74). 


Table 6. Results of teachers’ perceptions of their adaptation: Context (N=12) 


Construct Number of Items — Cronbach’s a M SD 
Admin Support 3 90 3.89 76 
Technical Support 3 .86 3.47 74 
Professional Development 3 76 4.11 .73 
Resources [Facility/Materials] 3 90 3.86 .90 
Total 12 62 3.83 39 


Teachers’ Challenges Implementing the New CS Curriculum 


Teachers challenges of implementing the CS curriculum were identified from open-ended questions in the 
four categories (1.e., teacher, student, curriculum, context), and these are discussed below. 

Teacher. The analysis of teachers’ open-ended responses indicated that they feel challenges in (1) lack of 
teaching methods for differentiate instruction, and (2) lack of CS content knowledge. Over 50% of teachers reported 
the level of programming skills of their students vary, and they have different grade level students in the same class. 
Some teachers struggled with addressing diverse students’ needs and they sought effective pedagogical approaches, 
specifically differentiated instruction to better engage students in CS learning. Some teachers examined: 


“Managing the speedy students who complete everything quickly and the slower students who 
struggle to complete their programs on-time” [Teacher 3]. 
“Having 6th, 7th, and 8th grade together in the same class....” [Teacher 6.] 


Although many teachers perceived they have received appropriate professional development, a few 
teachers felt a lack of CS content knowledge and sought further training, specifically on the topic of HTML and 
CSS. For instance, one teacher stated: “For me it will be the HTML and CSS until I can fully understand it” 
[Teacher 10]. 

Curriculum. Although it was not frequently mentioned, a few teachers expressed concern regarding 
assessing students’ learning outcomes. For example, when asked about teachers’ challenges implementing the CS 
curriculum, one teacher mentioned: “Assessing students' computational thinking skills” [Teacher 3]. 

Student. In the student category, our analysis revealed that teachers were struggling in teaching CS due to 
their students’ lack of (1) basic computer skills, and (2) motivation and engagement. Most teachers mentioned that 
many students have no basic computer skills. For example, one teacher stated that: “Some students have NO 
experience working with a computer at all....” [Teacher 6]. Some teachers also reported that their students are not 
motivated and not engaged in their learning. 

Context. In context, approximately 40% of teachers reported they do not have enough headphones and have 
network issues in their classroom. For instance, one teachers explained, “We need new computers, a class set of 
headphones, fewer restrictions on student accounts, and better server setup for more consistent access to online 
software” [Teacher 1]. 


Discussion 


This study investigated the current status of the adoption of the newly CS curriculum in middle schools, as 
well as the challenges that teachers faced in teaching CS. Based on our findings, we proposed some suggestions for 
implementing CS curriculum. 

First, teacher competency development needs to be enhanced. The findings showed that although many 
teachers perceived that they received sufficient professional development opportunities, some teachers commented 
that they need additional training on certain topics such as HTML and CSS. Because some teachers may not have 
been adequately prepared to acquire the contents knowledge and teach the topic. Thus, regular professional 
development sessions should be offered to CS teachers based on the teachers’ specific needs. In addition, further 
support needs to be provided to foster effective teaching practices. Result indicated that teachers face difficulties to 
better engage their students in learning activities due to students’ mixed abilities in one classroom. Many teachers 
also reported that students lack motivation and engagement. To enhance students’ learning experience in CS 
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classrooms, we could encourage teachers to develop CS-specific instructional strategies, especially differentiated 
instruction, via professional networks or local-based communities, which may facilitate conversations among 
teachers, researchers, practitioners and administrators. There are number of studies on case-based learning indicating 
that reviewing other teachers’ teaching practices using videos may enhance teaching competency (Gamoran & van 
Es, 2002; Seidel, Blomberg & Renkl, 2013). Thus, CS teachers could receive the same benefits by reviewing 
exemplary teaching practices for their areas. The examples could include a developed activity, or video cases 
capturing classroom practices. Authentic examples will provide teachers with more valuable contextual knowledge 
and support for adopting the curriculum in their context. For this, multiple exemplary materials and resources related 
to teacher training should be developed in CS education. 

Second, regarding curriculum, we should revisit the scope and sequence of curriculum, and revised it based 
on gaps identified in our analysis. The results of this study indicated that many students lack basic computer skills 
and programming experience. To promote students’ basic understanding of the content and practices, teachers used 
their instruction time, and as a result, they could not complete the CS activities as they were originally designed. 
Building students’ basic computer skills, which are required to learn CS, could be addressed in advance in other 
courses, such as Technology Application, or elementary school level. Our findings showed that the “game design” 
module was both students’ favorite and most challenging module in the curriculum. This implies that its complexity 
and entertaining content may facilitate students’ engagement and motivation, and, at the same time, challenge them 
to complete the activity. Accordingly, to better engage students and facilitate their problem-solving processes, 
scaffolds can be designed in advance in anticipation of students’ difficulties (Saye & Brush, 2002). Thus, to 
facilitate students’ motivation and engagement, additional scaffolds can be designed and embedded in the activity. 
With respect to differentiated instruction, the current curriculum needs to be redesigned to support these teachers’ 
needs and enable them to customize the curriculum in each school. 

Lastly, regarding context, school and district administration needs to provide on-going support, especially 
infrastructure. The results of this study suggested that some schools may not have appropriate equipment or 
facilities, such as headphones, software, or networks, which hinder efforts to offer a high-quality of CS education. 

The findings of this study provide useful insights and understanding about the implementation of CS 
curriculum with respects to four specific areas. The results can be used for building common shared goals among 
teachers, students and their parents, and school and district administrators to strengthen the new CS education 
initiatives. 

Based on our findings, further research needs to be conducted on several research topics. First, more in- 
depth study to evaluate the newly developed CS curriculum implementation will help to identify areas of 
improvement. Conducting a case study based on teachers’ and students’ interviews, as well as observations may 
bring deeper understanding of individual schools’ contexts. Second, rigorous measurements which can assess 
students’ content mastery and computational thinking skills should be developed. Our findings imply that teachers 
experience difficulty in developing assessments. Furthermore, to improve the current CS curriculum, students’ 
learning outcomes and performance should be identified and reflected in the revision of CS curriculum in the future. 
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Abstract 


Instructional designers often provide leadership to their institutions and organizations, especially with 
respect to creating effective learning content, systems, and environments. This quantitative experiment provides 
empirical, research-based evidence regarding the impact of various video resolutions and camera angles on student 
perceptions of subject matter credibility and effective immediacy of instructors. Feedback from randomly assigned 
participants indicated that instructor credibility and immediacy increased when the instructor was recorded and 
presented using eye-level, high-definition cameras as compared to lower resolution cameras located above eye 
level. The findings suggest that credibility and immediacy can be enhanced when instructors are able to make 
virtual eye contact with learners in multimedia designs and presentations. Instructors, instructional designers, and e- 
learning administrators can use these findings to improve multimedia presentations, especially with respect to the 
use of audiovisual techniques in online and e-learning programs. 


Introduction 


This project investigated how the instructional message design elements of video resolution and video 
recording angle of an online instructor affects student perceptions of instructor credibility and immediacy. This 
study builds on previous research into multimedia learning theory by filling several gaps in the instructional design 
knowledge base. Multimedia learning theory describes and applies cognitive load theory, dual-coding theory, and 
working memory models to define specific heuristics that can be used to effectively present content to learners 
(Mayer, 2009; 2014). Students learn more efficiently when they process integrated visual and auditory 
content. Similarly, motivation and learning are affected by other factors such as instructor credibility and 
immediacy in multimedia environments that include instructor presence or recordings (Jayasinghe, Morrison, & 
Ross, 1997). Credibility is the learners’ perception of an instructor’s genuine concern for their students, subject 
matter knowledge, and ability to be an effective teacher (Myers & Martin, 2006). Immediacy is the learners’ 
perception of the presenter’s verbal and nonverbal communication skills (McCroskey, J., Richmond, & McCroskey, 
L., 2006). Based on this previous research, it appears that both credibility and immediacy variables can be 
positively or negatively impacted by message design, especially multimedia designs that integrate video imagery of 
the instructor. 
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Research Questions 


The specific purpose of this research project was to study the impact of variations of video resolution and 
camera angle during the creation and delivery of audio and video multimedia modules for undergraduate and 
graduate students in online classes at a mid-sized, public, metropolitan university: 


RQI1: What is the effect of high-resolution and eye-level camera angle, low-resolution and eye-level camera 
angle, high-resolution and high-camera angle, and low-resolution and high-camera angle on perceptions of 
instructor credibility? 


RQ2: What is the effect of high-resolution and eye-level camera angle, low-resolution and eye-level camera 
angle, high-resolution and high-camera angle, and low-resolution and high-camera angle on perceptions of 
instructor immediacy? 


Research Design 


One hundred and eight student volunteers were randomly assigned into one of four treatment groups, 
resulting in at least 25 participants in each of the four trials. The consistent instructional content used during each 
treatment was designed specifically to be part of an online teaching with technology course, wherein, the instructor 
gave a brief mini-lecture on the history of social networking. The instructor’s authentic 20-minute audiovisual 
module for their online e-learning course was recorded from two cameras (eye-level or above) and was viewed by 
participants in one of two video resolutions (low or high). The combination of these two variables resulted in four 
treatment groups that included a high-resolution and eye-level camera angle, a low-resolution and eye-level camera 
angle, a high-resolution and high-camera angle, and a low-resolution and high-camera angle group. The first camera 
used to record the presentation was located at eye level with the instructor. The second camera was located 15- 
degrees above eye level to simulate being mounted higher on a classroom wall and closer to the ceiling. The videos 
were presented to student participants in either 1920x1080, high-definition or a lower, 320x240 resolution. The 
result was four variations of the same multimedia presentation where the instructor, content, and audio narration 
were identical with only the video resolution and camera angle changed between groups, see Figure 1. 
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1. High-resolution, Eye-level camera 2. Low-resolution, Eye-level camera 


3. High-resolution, High-angle camera 4. Low-resolution, High-angle camera 


Figure I. Each treatment group viewed one of four variations of the same multimedia presentation, 
though the presented video resolution and angle of recording camera varied. 


All four independent trials viewed the video in their specific treatment group on the same classroom 
displays and in the same classroom to remove extraneous variables related to display type and size. Once students 
viewed the presentation in their group they next completed two identical surveys. The McCroskey Source 
Credibility Measure, an 18-item, seven-option Likert-scale instrument with a Cronbach’s alpha reliability of .94, 
was used to measure instructor credibility (McCroskey & Teven, 1999). The Anderson Perceived General 
Immediacy Scale, a nine-item, seven-option Likert scale instrument with a Cronbach’s alpha reliability of .96, was 
used to measure participants’ perception of the instructor’s immediacy (Anderson, 1979). 
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Results 


A series of Analysis of Variances (ANOVAs) were used to evaluate the impact of video resolution and 
camera angle on instructor credibility and immediacy. McCroskey’s Source Credibility Measure resulted in a 
statistically significant difference between the treatment groups. A 4x1 analysis of variance (ANOVA) comparing 
the four groups indicated significant differences, F(3,104) = 3.34, p = .022, between viewers of the high-resolution 
and eye-level camera angle, low-resolution and eye-level camera angle, high-resolution and high-camera angle, and 
low-resolution and high-camera angle versions. A follow-up Tukey post hoc analysis indicated that participants 
who viewed the high-resolution, eye-level recording rated the presenter as more credible, especially in terms of the 
goodwill component of the construct (M = 4.93, SD = 1.12), as compared to participants viewing the low-resolution, 
high-angle recording (M = 4.2, SD = .89), see Figure 2. 


Instructor Credibility 
=" 4x1 ANOVA: F (3,104) = 3.34, p = .022 


M=4.84 
SD=.92 


M=4.44 


SD=1 
M=4.2 


SD=.89 


3.8 


High-resolution + Low-resolution + High-resolution + Low-resolution + 
Eye-level camera* Eye-level camera  High-angle camera High-angle camera* 


Figure 2. Instructor credibility: An ANOVA and Tukey post-hoc analysis indicated significant differences between 
the high-resolution and eye-level recording as compared to the low-resolution and high-camera angle recording. 


Also, Anderson’s Perceived General Immediacy Scale revealed a statistically significant difference 
between the treatment groups. A 4x1 ANOVA comparing the four groups indicated significant differences, 
F(3,104) = 3.56, p = .017, between viewers of high-resolution and eye-level camera angle, low-resolution and eye- 
level camera angle, high-resolution and high-camera angle, and low-resolution and high-camera angle versions. A 
follow-up Tukey post hoc analysis indicated that participants who viewed the high-resolution, eye-level recording 
and the low-resolution, eye-level camera recording rated the presenter higher in communication immediacy (M = 
5.28, SD = 1.35 and M = 5.25, SD = 1.27 respectively) as compared to participants viewing the low-resolution, 
high-angle recording (M = 4.27, SD = 1.56), see Figure 3. 
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Instructor Immediacy 
= 4x1 ANOVA: F (3,104) = 3.56, p = .017 


5.5 
M=5.28 
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Eye-level camera* Eye-level camera* —_High-angle camera = High-angle camera* 


Figure 3. Instructor immediacy: An ANOVA and Tukey post-hoc analysis indicated significant differences between 
the high-resolution and eye-level and the low-resolution and eye-level recordings as compared to the low-resolution 
and high-camera angle recording. 


Conclusions 


The findings of this study suggest that camera angle and video resolution play a role in enhancing student 
perceptions in audiovisual modules created for e-learning programs. The high-resolution versions appeared to 
communicate subtler nonverbal cues than the low-resolution versions. Similarly, the eye-level versions appeared to 
enhance the communication effectiveness of the instructor and reduced participants’ sense of distance. Designers 
can use these findings to define best practices and advocate that online instructor credibility and immediacy are 
enhanced when audiovisual presentations are recorded from eye-level cameras and presented in high resolutions. 

Investments in video production and content distribution are often expensive and time intensive. However, 
using effective recording resolution and camera angles can have a positive and influential impact on student 
satisfaction and improve the effectiveness of message design in multimedia-based e-learning programs. This present 
study also creates a baseline for future projects. Follow-up research will include presentation slides, additional 
video content, real-time telepresence, and more diverse subject content to further explore the impact and application 
of these variables on multimedia learning theory. 
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Abstract 


This study was designed to investigate how representation tool types, designated “visible annotation”, 
facilitate the learning process and enhance learning outcomes in CSCL environments. Twenty-eight students 
participated in this study, and they were divided into two groups: the TLL-type focusing on defining the concept of 
learning contents and the CTLL-type focusing on discussing the concept of learning contents. The result indicated 
that TLL-type was more effective in both the learning process and learning outcome. 


Introduction 


Computer-supported collaborative learning (CSCL) is a learning method that facilitates collaborative 
learning using computers and networked devices (Schwarz, Groot, Mavrikis, & Dragon, 2015; Stahl, 2015). Some 
CSCL researchers have reported that CSCL is more effective than face-to-face learning because it can help learners 
overcome the limits of physical space using computational devices (Fischer, Bruhn, Grasel, & Mandl, 2002; Pena- 
Schaff & Nicholls, 2004; Schellens & Valcke, 2006). Learners carry out complex task through sharing each other’s 
knowledge, negotiating various opinions, and integrating different perspectives in CSCL (Eryilmaz, van der Pol, 
Ryan, Clark, & Mary, 2013). It is effective for enhancing individuals’ knowledge construction and achieving a 
higher-level of collaborative knowledge construction (Garrison & Arbaugh, 2007; Morgan, Whorton, & Gunsalus, 
2000). However, in the process of achieving higher-quality solutions, many restrictions can be imposed due to the 
learners’ diverse perspectives in such areas as sharing problem awareness, negotiating a variety of opinions, and 
building collaborative knowledge based on communicative activities (Fuks, Pimantel, & Lucena, 2006). In addition, 
collaboration load can occur in CSCL environments, and this can lead to ineffective learning processes and 
unsuccessful learning performance. Most previous research showed that representation tool can support learners’ 
construction of accurate shared knowledge and a higher level of constructed knowledge focusing on learning 
outcomes (Fuks et al., 2006). However, few efforts explored the impact of learning performance on the sequential 
process of knowledge construction requiring various interaction types according to the nature of the complex task. 

This study was designed to investigate how representation tool types, designated “visible annotation,” 
facilitate the learning process and enhance learning outcomes in CSCL environments. Based on the principle of 
collaborative knowledge construction, we developed the following two types of visible annotations with a linked 
annotation function: the TLL-type, representing the one meaning-sharing learning phase focusing on defining 
concept of learning contents and one opiniont-sharing learning phase used to build shared knowledge (TLL); and the 
CTLL-type, representing the one meaning-sharing learning phase focusing on discussing concept of learning 
contents and one opinion-sharing learning phase used to build shared knowledge (CTLL). The two visible- 
annotation types include the same problem-solving learning phases used to build constructed knowledge. 
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Specifically, the following research questions were addressed in this study. First, what are the effects of the 
visible-annotation tool on the learning process in the CSCL environment? Second, what are the effects of the 
visible-annotation tool on the learning outcome in the CSCL environment? 


Theoretical Background 


Collaborative learning is effectively carried out if there are active interactions between team members in 
CSCL environment (Kolloffel, Eysing, & de Jong, 2011). However, if the number of unshared messages among the 
learners grows, collaborative interaction is replaced by simple interaction, and this does not positively influence 
shared knowledge building and collaborative learning outcomes (Suthers, Vatrapu, Medina, Joseph, & Dwyer, 2008). 
To facilitate meaningful interaction in CSCL, researchers have designed and developed a variety of tools (Kolloffel 
et al., 2011). Especially, many strategies using representation tools have been verified that specific functions (e.g., 
concept map, graph, matrix, and annotation) can induce active interaction within learners and acquire high-quality of 
learning outcomes by visualizing learners’ cognitive process (Bell, 1997; Bruggen, Kirschner, & Jochems, 2002; 
Schwarz et al., 2015). Recently, Eryilmaz et al. (2013) showed that linked-annotation function based on 
asynchronous discussion can facilitate interactions of conflict and assertion, which is related to the effects of 
supporting the “negotiation of position” phase (Beers, Kirschner, Boshuizen, & Gijselaers, 2005, p. 10). However, 
they fail to verify to effects of supporting the “negotiation of meaning” phase such as interactions of clarification, 
interpretation (Beers et al., 2005, p. 10). 

As Beers et al. (2005) and Rummel and Spada (2005) indicated, meaningful interaction in CSCL requires a 
series of phase from unshared knowledge to collaborative knowledge through several sharing activities. In particular, 
the optimal learning methods for each learning phase should be applied when carrying out the sequential sharing 
activities of learning content from the key concept to more complex learning content (Cannon-Bowers & Salas, 
2001). The exact meaning sharing of learning content in an ill-structured task can help learners reduce the 
possibilities for misunderstanding and lead them to in-depth discussions (Slof, Erkens, & Kirschner, 2011). 
Considering the importance of exact meaning sharing in a knowledge construction process for the complex task 
(Beers et al., 2005; Jorczak, 2011; Rummel & Spada, 2005), Shin, Kim & Jung (in press) suggested the 
representation tool adapting different types of represenation function. Specifically, learning process consisted of 
three phases based on nature of part-tasks and the process of collaborative knowledge construction; meaning 
sharing, opinion sharing, and problem-solving. The result revealed the meaningful effectiveness on the accuracy of 
sharing activities and level of collaborative performance (Shin et al., in press). However, the finding has still a 
limitation because it could not deeply catch interaction patterns occurring each sharing phase (Fu, Aslst & Chan, 
2016; Yiicel & Usluel, 2016). 

Although recent researches have been conducted to analyze qualitative dialogue, they rarely analyzed 
results systematically by considering theoretical principle (Fuerstenau, Ryssel, & Kunath, 2010; Schellens & 
Valcke, 2005; Yiicel & Usluel, 2016). The recent research of the analyzing discussion patterns has reported that 
much research has shown a noticeable problem that fails to consider interaction pattern both theoretically and 
practically to promote various collaborative interaction activities (Fu et al., 2016). To overcome this limitation, the 
present study aims to deeply explore the sequential knowledge construction process by using representation tool 
reflecting knowledge construction principle. 


Method 


To test the hypotheses, this study analyzed the online messages posted on an asynchronous discussion tool 
(Visible-annotation) that was used as a learning environment. As a reflecting limitation of previous researches, this 
paper focused on the analysis of the knowledge construction process more deeply. Moreover, this study tries to 
explore the process of interaction in relation to the collaborative learning outcome. 


Procedure 


The experiment was conducted as a three sub-tasks for four weeks in an online learning environment 
without a face-to-face meeting. The participants first performed pre-test which consisted to prior knowledge, 
computer literacy, collaborative tendency, and was provided with the manual included to learning tasks and method 
of using the tool. The students participated in pairs in the experiment. All of the groups performed sub-tasks through 
same material related to educational methodology. However, there was a difference of annotation type. Specifically, 
sub-tasks consisted of core-concept understanding and sharing of learning material, specific-content sharing of 
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learning material, and lesson planning task. Students were asked to submit a lesson planning in pairs at the end of 
collaborative learning (see Figure /). 


2) Meaning 3) Opinion 4) Problem 5) Collaborative 


1) Pretests sharing sharing solving outcome test 


Figure I. Learning processes 
The Compared Tools for Analyzing Other Treatment 
To compare the effectiveness of treatment, the different two types of visible annotations were used. The 
tool of CTLL type was considered managing strategy for the type of tool, which has two times sharing activities 


through sentence-based annotation only (see Table 1). 


Table 1. Learning phases and learning methods based on visible annotation types 


Visible Sharing phase Problem-solving phase 
annotation Comprehension of learning content task Lesson planning task 
type 1 week 2™ week 34 & 4th week 
Meaning sharing learning Opinion sharing learning phase Problem-solving learning 
phase 1) Sub-task phase 
1) Sub-task - To ask, submit of opinion 1) Sub-task 
- To define the meaning and and comment on specific - To negotiate various 
CTLL explain of core-concept of content of material opinions and derive 
material 2) Function of tool solutions for 
2) Function of tool - Sentence-based annotation completing the lesson- 
- Sentence-based - Linked annotation planning task 
annotation 2) Function of tool 
-__ Linked annotation - Sentence-based 
Meaning sharing learning Opinion sharing learning phase annotation 
phase 1) Sub-task - Linked annotation 
1) Sub-task - To ask, submit of opinion 
- To define the meaning and and comment of specific 
TLL explain the pros & cons of content of material 
core-concept of material 2) Function of tool 
2) Function of tool - Sentence-based annotation 
- Word-based annotation - Linked annotation 
- Linked annotation 


Interaction Analysis of Knowledge Construction Activities 

We investigated the frequency and proportion of discussion activities in each group. The messages were 
analyzed by Pena-Shaff and Nicholls (2004) sentence analysis tool. Three educational experts reviewed and used 
content validity. The level of inter-rater reliability was .78. 
Analysis of Collaborative Learning Outcome 

The learning outcome was measured by the lesson-planning task. A three-point Likert scale was developed 
to measure the level of the constructed knowledge, based on the research of Dick, Carey, and Carey(2003), and 
Gagné, Wager, Golas, and Keller (2005).The level of inter-rater reliability of three evaluators is .89. 

Results 
This study carried out the sequential analysis based on the quantitative content analysis results. Annotations 


coded by the scheme were organized chronologically. The circles in following figures represent knowledge 
construction activities. The arrows between circles depict directed transitions. The thickness of an arrow is 
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proportional to the probability of a transition. The values specified in the figures illustrate transitional probabilities. 
To ease readability, transitional probabilities less than 0.30 were omitted in the figure. Also, we investigate the z- 
score which take into account not only the observed total number of responses to a particular messages category but 
also the marginal totals of each response type observed across all message types (Jeong & Frazier, 2008). 


Meaning Sharing Phase 


First, in the CTLL type, sequential analysis results of interactions in meaning sharing phase showed a 
pattern that almost activities such as interpretation, question, assertion, and support were converged to clarification. 
However, the result of z-score was only statistically significant in patterns from clarification to clarification (z = 
2.78, p <.01). On the other hand, despite the small cell frequencies, there was significantly higher than expected 
from conflict to question (z = 7.84, p <.01) (see Figure 2). Next, in the TLL type, all seven activities in meaning 
sharing phase were converged to clarification activities. And there were statistical significant results of z-score from 
clarification to clarification (z = 2.78, p <.01), from Interpretation to interpretation (z = 9.19, p <.01), from question 
to clarification (z = 1.81, p <.05), and from assertion to clarification (z = 2.19, p < .05) (see Figure 3). The 
interaction patterns of two groups were similar to those of clarification, but TLL type was differed in that they have 
significant z-score with various activities. In the case of CTLL type, a pattern from conflict to question, which is 
expected to be derived from the opinion sharing phase, was derived at the meaning sharing stage. 
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Figure 2. Transitional state diagram for meaning sharing phase of the CTLL type 


89 


CONSENSUS 
| INTEPRETATION \ BUILDING 


QUESTION 


a4 CLARIFIACTION 
| SUPPORT 


Figure 3. Transitional state diagram for meaning sharing phase of the TLL type 


| ASSERTION 


a 


Opinion Sharing Phase 


First, in the CTLL type, the sequential results of interaction showed a pattern that converged to the 
assertion. Moreover, z-score of sequential analysis was only statistically significant in patterns from assertion to 
assertion (z = 5.38, p < .01). On the other hand, there was significant result from clarification to clarification (z = 
7.55, p <.01), and from interpretation to interpretation (z = 5.90, p <.01) (see Figure 4). Similar to results of the 
CTLL type, the sequential pattern in TLL type was also converged to assertion activities. Specifically, there were 
Statistical significant result of z-score from assertion to assertion (z = 3.14, p < .01), from clarification to 
interpretation (z = 4.43, p < .01), and from interpretation to interpretation (z = 4.66, p < .01) (see Figure 5). In 
conclusion, in the opinion sharing phase, two groups had similar patterns to the interaction, unlike the meaning 
sharing. 
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Figure 4. Transitional state diagram for opinion sharing phase of the CTLL type 
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Figure 5. Transitional state diagram for opinion sharing phase of the TLL type 


Effects of Types of Tool in Learning on Collaborative Learning Outcomes 


The results of ANCOVA of the constructed knowledge found significant differences between conditions /F 
(1, 23) = 8.59, p = .008]. The level of collaborative learning outcome in the TLL type condition (M=10.86, 
SD=1,49) was higher than the level in the CTLL type condition (V=8.04, SD=2,72) (see Table 2). The results 
implied that the concept sharing phase of the learning content could affect the level of the collaborative learning 
outcome. 


Table 2. Effects of types of tool on collaborative learning outcomes measures 


Types of tool M SD AM 


CTLL 8.04 2.72 8.07 
TLL 10.86 1.49 10.83 


N=28. M=Mean. SD=Standard deviation. AM=Adjusted mean. 
Conclusions 


The results were consistent with the empirical evidence for the knowledge construction principle previously 
obtained in CSCL research (Beers et al., 2005; Rummel & Spada, 2005). The TLL focusing on concept learning was 
found to more effective in learning process and learning outcome. Specifically, TLL-type facilitates clarification and 
interpretation activities for help understanding about learning content, and these activities lead to assertion activities 
that enhance the quality of ill-structured problem-solving. However, lower activities occurred in TLL than CTLL, 
such as consensus and conflict activities, should be explored with re-analysis. 
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Abstract 


Students should use the internet responsibly. Teachers, and school IT specialists have roles in keeping 
students safe. This study aims to determine the readiness levels of preservice ICT teachers about online treats. A 
unique realistic problem was provided to pre-service ICT teachers; and asked for solution strategies. Preservice ICT 
teachers’ solution strategies and suggested activities were analyzed using content analysis method. Eight categories, 
which were education, technical solutions, guidance, parents, problem definitions, obstacles, activities, and 
cooperation emerged from the analysis. Results showed that most of the pre-service teachers were not as ready as 
desired. Although they knew the importance of education and training for internet safety and security they couldn’t 
emphasize the importance of integrating internet safety education into the curriculum. Also they did not seem to be 
eager to take responsibility about students’ internet safety. 


Keywords: internet safety, pre-service ICT teachers, readiness levels of preservice ICT teachers 
Introduction 


The appropriate and safe use of internet requires safe keeping of personal information and not sharing them 
in public media. However, most people are not aware of this problem. A study conducted as a part of the EU Kids 
Online project revealed that 42% of the children and teenagers in Turkey have public profiles on social media open 
to everyone and only third of the children and teenagers share their profile with their friends (Karakus, Cagiltay, 
Kasikci, Kursun, & Ogan, 2014). According to researchers people who share private information on internet risk of 
privacy violations or coercive or hostile interactions more than people who do not (Elisabeth, S., & Livingstone, S. 
2016; Staksrud, Olafsson, & Livingstone, 2013). Experts suggested that students should not share their passwords, 
photos, personal information and identification information on social media and control their privacy settings often. 
According to Staksrud et al. (2013) children who use social networking sites (SNS) are more likely to have received 
sexual messages, seen sexual images on websites, encountered negative user generated content, been bullied on the 
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internet and met people offline that they had only met online before. And these are not the only dangers children can 
experience online. 

Some of the dangers of sharing personal information on the internet are: internet fraud, threat, harassment, 
financial loss, physical, social, and psychological injures. The report by the International Telecommunication Union 
(2013), listed the major dangers experienced by the children online as, pornography, violence, online gaming 
addiction, cyber fraud, cyberbullying, and racism. Children can be exposed to illegal content, or legal but age 
inappropriate content. They can be contacted by sexual predators, or participate to harmful online communities such 
as sites which encourage anorexia, self-harm or suicide. They can engage in risky sexual interactions such as sexting 
or place information about themselves in the public domain, or post pictures or videos or text which might 
compromise their personal safety or jeopardize a number of career options for them in the future. Children can be 
exposed to bullying and to find themselves in an environment that allow or promote bullying others. Children can 
access or acquire age inappropriate goods and services. Children and young people can be target to scams, identity 
theft, fraud and similar threats which are economic in nature or are rooted in inadequate data protection or privacy 
laws. With some children and young people, forms of obsessive behavior or excessive use can be a big problem. 
Internet addiction, game addiction and similar problems can occur. 

Children’s exposure to these dangers can turn into personal and social problems. Fear, sadness, hate, 
disappointment, shame, worry, turning to violence, self-mutilation, sleep and eating disorders, failing at school and 
social relations, problematic behavior towards family members and friends are some of these problems. 

Students should use the internet responsibly, they should only share safe content, and request help from an 
adult they trust when they experience any issue. Also, parents, teachers, school administrators have roles such as, 
informing student safe use of internet, taking precautions towards ill-treatment students can experience online, and 
intervening with proper methods when students experience a problem. However students engage in dangerous online 
activities such as trusting strangers and chatting with them online, and disseminating personal information online 
from early age (Kavuk, Keser, & Teker, 2011). 

Also, children and teenagers are using internet more and more. According Turkish Adolescent Profile 
Report (2014), computers, tablets and smart phones are being used by children and teenagers between the ages of 12 
and 15 at an overwhelming rate. 65 % of Turkish adolescents have computers at their homes and 48 % have mobile 
phones and half of these are smart phones. 82 % of the homes with computers have also internet access. 
Socioeconomic (SES) status and computer and smart phone ownership correlate positively. At high SES computer 
ownership rate increases to 96 %. 47 % of these adolescents with access to internet do it without any restrictions. 
Half (48 %) of the adolescents with internet access have online friends that they communicate regularly. 29 % of the 
adolescents moved their friendships to face to face settings after meeting them online. Adolescents use the social 
networking sites at an awe-inspiring rate of 76.5 %. Access to internet and smartphones without any oversight and 
guidance can put children at risk. Similarly access to and use of internet via mobile devices by children and 
teenagers are increasing and will likely be an issue in the future all over the world (Stald, Green, Barbovski, 
Haddon, Mascheroni, Sagvari, Tsaliki, 2014). 

Previous studies show that, both teachers and students see teachers capable of recognizing, taking 
precautions and intervening to internet related safety dangers (Kavuk, 2016). Teachers see themselves and students 
see their teachers capable of recognizing, preventing and intervening cyberbullying related cases. Although teachers 
see themselves capable they rarely intervene to online problems their students face. Experts suggested that teachers 
were afraid of overstepping the legal boundaries (Kavuk, 2016). 

Teachers, administrators, parents, school IT specialists and students together create an action plan to 
prevent and intervene online threats students can experience. Information and Computer Technology (ICT) Teachers 
are the specialist in Turkish schools. ICT teachers would be expected to fulfill this role in addition to regular 
teachers’ role. There were not any studies about ICT teachers’ readiness towards fulfilling this role as a whole. 
There were studies on identifying preservice teachers’ perceived awareness of and preparedness for cyberbullying 
cases (Li, 2008; Ryan & Kariuki, 2011; Yilmaz, 2010). In Canada, Li found that preservice educators did not feel 
qualified (prepared) in the identification or management of cyberbullying. Similarly Ryan and Kartuki (2011) found 
that half of the preservice teachers see themselves as not ready to handle cyberbullying cases. Yilmaz (2010) 
conducted the same study in Turkey and found slightly higher perception levels however still half of the students 
were not confident about their perceived abilities to handle cyberbullying cases. Since cyberbullying is not the only 
problem preservice teachers should be able to handle diverse sets of online threats and cyberbullying research 
clearly shows there needs to be more direct approach to measure preservice teachers readiness levels. This study 
aims to determine the readiness levels of preservice ICT teachers towards determining, preventing, and intervening 
online treats towards students. 
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Methodology 


This case study was conducted with 38 preservice ICT teachers who were senior students at a public 
University in Turkey Students age ranged between 21 and 28. Qualitative data was collected to assess their readiness 
levels. A unique realistic problem was provided to ICT preservice teachers: 

Students at the local public middle school registered to a fishy Illuminati themed web site using 

their personal information. The students started to think that they were being followed by this so 

called Illuminati organization and were in danger because of the rumors they started themselves 

and these turned into urban legends at the school. The situation became widespread among the 

students who registered to the website and started to cause behavioral and psychological problems. 

How would you solve this problem if you were the ICT teachers at this school? 


Participants were asked for solution strategies and activities to assess whether they have necessary skills to 
cope with these kinds of problems in real life. Problems was a mass hysteria like problem in a middle school setting 
about a real web site masquerading as an official historical organization named Illuminati; and they wrote down 
their solutions for this problem. Content analysis method was used to analyze this qualitative data. 


Findings 


Eight categories were proposed based on the solutions participants offered. These categories were 
education, technical solutions, guidance, parents, problem definitions, obstacles, activities, and cooperation. 
Strategies including training and information for students were categorized into education category. Strategies such 
as technical access control and blocking were categorized in to technical solutions category. Strategies including use 
of guidance services and psychological support systems were categorized as guidance. Training and education of 
parents were categorized as parents. Suggestions about the nature of the problem were categorized under the 
definition of the problem label. Suggestions regarding unavoidable nature of the problem and real life challenges 
were categorized under obstacles. Strategies trying to take advantage of indirect social, sportive and cultural 
activities categorized under activities label. Strategies that suggested cooperation between stakeholder such as 
parents, teachers, and administrators were categorized under cooperation label. 


For education category some of the participants’ suggestions were: 

“Educate students about the faultiness of disseminating the wrong information and conduct awareness 
activities about information online”. 

“Digital literacy training should be given.” 

“Seminars related to the subject can be given.” 

“Courses by experts should be provided.” 

“Banners and posters can be used.” 

“Conference for the parents.” 

These suggestions and activities were all educational or informative activities. For technical solutions 
category some of the participants’ suggestions were: 

“Access restriction.” 

“Ban the sites.” 

“ICT ban.” 

“Internet filter (restriction) in school.” 

“Internet access should be restricted both at school and at home.” 

All of these suggestions were basically same and involved blocking access to the problem site. For 
guidance category some of the participants’ suggestions were: 

“Individuals directed to the guidance center” 

“Student monitoring” 

“Guidance services should be used.” 

“Developmental needs to be addressed.” 

“Using group guidance services.” 

Guidance category suggestions mainly about utilizing the school guidance counselor services. For parents 
category some of the participants’ suggestions were: 

“Parents must supervise their children” 

“Parents should be informed about the problem.” 
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Parent category suggestions were about involvement. For problem definitions category some of the 
participants’ suggestions were: 

“Tluminati site is a fraud site” 

“First the situation should be analyzed.” 

“Students internet activities should be observed.” 

“A scientific study should be conducted about the problem.” 

“To understand the problem data should be collected.” 

Problem definitions category comments were about determining and describing the problem to initiate 
further activities. For obstacles category some of the participants’ suggestions were: 

“Simple ban or blocking would not solve the problem.” 

“The students are in a critical age and the consequence can be severe.” 

“Students tend to trust the information on the internet.” 

“The teacher should be informed about the problem.” 

“There is not a simple solution” 

Obstacles category comments were about complexities and difficulties of the problem. For activities 
category some of the participants’ suggestions were: 

“Students should be directed to extracurricular activities.” 

“Extracurricular activities should be conducted.” 

Activities category comments were about students partaking in extracurricular activities. For cooperation 
category some of the participants’ suggestions were: 

“Parents should be informed.” 

“Teachers should work with school administrators and parents.” 

“Students should be monitored not only at schools but also at home.” 

Cooperation category suggestions were about stakeholders working together to solve the problem. 


Results and Recommendations 


Results showed that most of the pre-service teachers were not as ready as desired. Although they knew the 
importance of education and training for internet safety and security they couldn’t emphasize the importance of 
integrating internet safety education into the curriculum. Their suggestions were mostly superficial suggesting about 
referring subjects to counselor or providing educational activities. Since the unique problem was mass hysteria type 
of problem, students could come up with more complex solutions. Although some of them were aware that simply 
banning the website wouldn’t solve the problem, their concern was more about students would still be able to access 
to the web site by other means. This problem might require teachers actually talking to students and understanding 
their concerns and provide support to them with all the stakeholders. Although few students suggested cooperation 
the debt of the answers were not at the level we need at our schools. Also, as information technology teachers, they 
should have felt responsible more; and not refer everything to counselors and administrators. There is definitely a 
room for academic improvement. 

There is a need for addition of internet safety related courses to teacher education curriculum and students 
should be provided real life cases to prepare them for school practice. 
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Abstract 


Cyberbullying is a serious problem among adolescents. Widespread use of digital communication tools is 
increasing the number of cyberbullying cases. Schools staff should be ready to overcome cyberbullying. This study 
aimed to determine the awareness levels of Turkish school about prevention of cyberbullying. Turkish educators’ 
awareness of creating anti-cyberbullying school climate, and instituting anti-cyberbullying technological measures 
was acceptable; but instituting anti-cyberbullying rules and regulations, and anti-cyberbullying educational actives 
were not acceptable. 


Keywords: cyberbullying, cyberbullying awareness, cyberbullying prevention 
Introduction 


Cyberbullying is defined as a willful and repeated harm inflicting acts, such as taunting, name calling, 
insulting, gossiping, knowingly spreading computer viruses, sharing someone’s photo without their knowledge, 
sending SMS messages from hidden numbers, threatening, harassing, and intimidating using information 
technologies. Considering the prevelance of cyberbullying (Wang, Iannotti and Nansel, 2009; Mishna, Kassabri, 
Gadalla and Daciuk, 2012; Erdur-Baker and Tanrikulu, 2010), its effects on individuals (Belsey, 2006; Beran and Li, 
2005; Hinduja and Patchin, 2005), the factors that trigger cyberbullying (Aftab, 2011; Bauman, 2013; Campbell, 
2005; Erdur-Baker and Tanrikulu, 2010;) and the legal sanctions, it is evident that cyberbullying is a serious threat 
that targets adolescents. This problem grows day by day and it causes concern among parents, educators, and school 
administrators. 

Researchers suggested that, most of the schools’ staff did not want to involve themselves in cyberbullying 
cases because they did not wanted to overstep legal boundaries (Willard, 2007). Moreover, most of the schools do 
not have preventive measures that educate and inform students about dangers and consequences of cyberbullying. 
The appropriate, ethical and healthy usage of Information Technologies (IT) should be supported, but at the same 
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time students, school staff and parents must be able to overcome cyberbullying issues that arise because of 
inappropriate use of IT. 

Jager, Amado, Matos, and Pessoa (2010) stated that cyberbullying is a unique type of bullying. For this 
reason, it is necessary that the studies of fighting with cyberbullying in schools are unique. Brewer (2011) defines 
cyberbullying as a challenge unlike any other that school leaders and other educators face. According to Huang and 
Chou (2013), teachers are in a critical position to cope with bullying. Bhat (2008) stated that the prevention and 
intervention measures for cyberbullying should be established together with school technology experts, managers, 
teachers, parents and students. 

Researchers make some suggestions about how to combat with cyberbullying. For example, Hinduja and 
Patchin (2009c) emphasized that the work to be done at school in order to prevent cyberbullying should include the 
dimensions of recognition, prevention and intervention. To recognize cyberbullying is to know what it is, how it 
occurs, its features like causes, effects or legal consequences, and to analyze the incidents that take place. To 
intervene effectively in cyberbullying, and clear school principals and rules of cyberbullying must be identified 
(Slonje and Smith, 2008) and the evidence and details the incidents must be examined. 

To prevent cyberbullying it is necessary to provide school-parent collaboration, to design an effective 
cyberbullying curriculum and to educate school stakeholders about cyberbullying (Cassidy, Brown and Jackson, 
2012). Farrington and Ttofi (2010) stated that effective sanctions and disciplinary methods would be useful in 
reducing cyberbullying. However, some researchers certainly do not suggest discipline and ignorance behavior 
(Bauman, Rigby and Hoppa, 2008; Yoon and Barton, 2008). DeSmet et al. (2015) suggests activities such as talking 
to students, working with parents and guidance specialists to cope with cyberbullying. Besides, some researchers 
emphasize the importance of creating positive school culture, educating school stakeholders, creating school specific 
principals and taking technological precautions within the scope of cyberbullying prevention studies (Cassidy et al. 
2012; Hinduja and Patchin, 2009a; Kowalski, Limber and Agatson, 2012). 

Since it is a serious problem targeting students, it is necessary to raise awareness about preventing 
cyberbullying to all school stakeholders, and this task falls primarily on school staff. In line with this, it is important 
to establish educators’ perception of cyberbullying, their reaction and current bullying experiences, and then, based 
on these factors, to devise prevention strategies (Huang and Chou, 2013). 

There were some resources and studies on evaluating the perceptions, opinions or awareness of educators 
regarding cyberbullying prevention, and creating specific cyberbullying strategies for the schools, in the literature 
(Hinduja and Patchin, 2009a; Kowalski et al., 2012; Mishna et al., 2006). However, studies have not been conducted 
to determine the opinions of educators in Turkey regarding cyberbullying prevention. For this reason, the main goal 
of this study was to determine the educators’ (e.g., school administrator, ICT teacher or school counselor) awareness 
regarding cyberbullying prevention. Following research questions was used to achieve this goal. 

1. What are the opinions of educators on creation of anti-cyberbullying school climate? 

2. What are the opinions of educators on construction of anti-cyberbullying principles? 

3. What are the opinions of educators on creation of anti-cyberbullying curriculum and education? 
4. What are the opinions of educators on taking anti-cyberbullying technological precautions? 


Methodology 


The study was conducted using survey method, and quantitative data was collected. The sample was 
selected among middle schools and high schools in Turkey. The population was 22249 schools. Stratified sampling 
technique was used to choose the sample schools. Using Cochran’s (1977) stratified sampling formula, the sample 
size of 378 was determined. Turkish Statistics Institutes’ (TSI) statistical region determination classification and 
provinces’ information technology crime rates was used. According to TSI’s classification, there are 12 regions in 
Turkey, and Istanbul province represented a region by itself. The province with the highest information technology 
crime rate and the province with the lowest information technology crime rate was selected from the remaining 11 
regions; 23 provinces in total included in the study. The study was conducted on 1431 schools; and 2586 educators 
who were employed in these schools at these provinces participated. If there was an ICT teachers and/or school 
counselors at these schools and at least one school administrator participated from each school. 57.1 % of the 
participants were school administrators, (f=1477), 20.1% were school counselors (f=521), 19.1% were ICT teachers 
(f=495), and 3.6% were other teachers (f=93). 

Data were collected using “Questionnaire for Cyberbullying Awareness at School” developed by (Keser 
and Kavuk, 2015). The questionnaire has 51 items in three dimensions, which are recognition, prevention, and 
intervention of cyberbullying. Prevention dimension had four sub dimensions: the creation of an anti-cyberbullying 
school culture, the creation of an anti-cyberbullying curriculum and education, the construction of specific anti- 
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cyberbullying principles, and taking technological precautions. In this study, the four sub dimensions and the 29 
items of prevention dimension were used for data collection. Each dimension and each item was analyzed 
separately. 

The data were collected in electronic form. Permissions and approvals for human subject research were 
granted by Ministry of Education and University Ethics Board. Schools were contacted via e-mail, phone and if 
possible with meetings and informed about the study and their participation to the online survey was requested. 
Survey submissions that met the ethical standards were analyzed. Data were analyzed using mainly frequency (f) 
and percentage. The answers given by the educators were accepted as clues to the readiness of the schools for 
preventing cyberbullying and the findings were discussed in this way. 


Findings 


Regarding the first research question, educators’ opinions on the creation of anti-cyberbullying school culture 
sub dimension were analyzed. Accordingly, 72.3 % of the educators reported that they believe students would report 
cyberbullying incidents to school staff; 82 % of the educators said that students were reminded regularly about 
reporting cyberbullying incidents; 68.3 % of the educators claimed that there were activates at their schools to show 
students that cyberbullying was not cool. The majority of educators (93.3%) stated students know that the inappropriate 
use of technology will not be ignored by the school management. 

For the second research question, educators’ opinions on the creation of anti-cyberbullying policies sub 
dimension were examined. Only 47.5% of the educators stated that their schools have a clear anti-cyberbullying policy; 
57.8 % of the educators reported that they had anti-cyberbullying rules and regulations; just 22.6% of the educators 
claimed that these rules and regulations were known by the parents; almost half of the educators gave negative response 
or stated that they had no information regarding these items. 

About the third research question, the creation of an anti-cyberbullying curriculum and education sub 
dimension were investigated. According to this, approximately 40% of educators stated that combating cyberbullying is 
not a part of the curriculum in the schools, or that they do not have knowledge on this subject. 65.5% of the educators 
thought teachers at their schools were adequate to handle cyberbullying, 50.7% of the educators thought the students at 
their schools were adequate to handle cyberbullying, and only 22.8% of the educators thought the parents of the 
students at their schools were adequate to handle cyberbullying. On the other hand; 43.3% of the educators reported 
that there were educational activities towards the students at their schools; 37.1 % of the educators reported that there 
were educational activities towards teachers at their schools; and 35% of the educators reported that there were 
educational activities towards the parents at their schools. However, 13.7% of the educators stated that the school staff 
does not organize any educational activities for students; 31.2% said there are no educational materials for teachers; 
and 40.4% believe there are no study materials for families. 78% of the educators claimed that school staff are aware of 
their responsibility for measures to be taken around the school to cope with cyberbullying. Most of the educators 
(73.8%) stated that they did not use peer mentoring in their schools to educate student about cyberbullying, or they 
have no idea on this issue. 60.9% of the educators think that students learn about cyberbullying from ICT teachers, 
58.7% from school administrators, 57.1% from school guidance teachers, 23.4% from family and 21.6% from friends. 

Regarding the fourth research question, educators’ opinions on the taking anti-cyberbullying technological 
precautions sub dimension were viewed. The majority of the educators reported that schools’ networks had internet 
monitoring (73.3%) and content monitoring (55.2%) software/hardware. 87.4% of the educators reported that they had 
antivirus software; and 87% of the educators reported that they were using government sanctioned safe internet service. 
95.1% of the educators declared that they avoid publishing the students’ personal information on the school website. 


Discussion 


Educators’ opinions on the first sub dimensions show that the schools’ level of readiness concerning 
creating anti-cyberbullying school culture is found to be generally high. Creating a school atmosphere where 
cyberbullying behaviors would never be ignored (Hinduja and Patchin, 2013) and creating a positive school culture 
in which cyberbullying is not popular among the student (Hinduja and Patchin, 2009b) is crucial in preventing 
cyberbullying. In our research, educators claimed they work to create such a school culture and environment. The 
vast majority of educators think that cyber victim students would report the incident to a teacher and they stated 
students are regularly reminded that they should seek help from teachers in such cases. Although similar results can 
be found in research conducted among educators, the situation is quite different from the students’ perspective. In 
Choucalas’s study (2013), the majority of the school administrators and teachers thought that students would report 
cyberbullying incidents, while nearly half of the students said they would report a cyberbullying incident. 
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The readiness level of the schools concerning the creation of anti-cyberbullying policies sub dimension is 
found to be generally low. Creating clear and strong anti-cyberbullying policies, principles and rules in schools is 
important to combat cyberbullying (Hinduja and Patchin, 2009a; Kennedy, Russom and Kevorkian, 2012; Kowalski 
et al., 2012; O'Moore and Minton, 2005). However, in our study, it can be concluded that the majority of the schools 
does not have a clear cyberbullying policy, principal or rule, or any policy or set of rules specific to cyberbullying. 
Although researchers emphasized that the rules and principals of traditional bullying do not work in cyberbullying 
(Shariff, 2005); in a similar study to ours, it is found that schools have no principles specific to cyberbullying; and 
educators have determined that the school bullying principles can also be used in cyberbullying incidents (Ryan, 
Kariuki and Yilmaz, 2011). 

The readiness level of the schools concerning the creation of an anti-cyberbullying curriculum and 
education sub dimension is also found to be generally low. Educators’ answers showed that cyberbullying is not 
included in in almost half of the schools’ curriculum or educational programs. Nevertheless, educators think that 
they are qualified to cope with cyberbullying. In the literature, there are studies that obtained similar results and 
obtained different results. For example, Choucalas’s study (2013), more than half of the teachers (52%) and more 
than two-thirds of the school administrators (70%) stated that the teachers knew how to recognize cyberbullying 
incidents. On the other hand, in another study, only 22.8% of the educators found themselves suited to cope with 
bullying (DeSmet et al., 2015). Schools have an important role in helping students, teachers, and families prepare for 
cyberbullying (Bauman et al., 2008; Belsey, 2006; Kennedy et al., 2012). Although educators find themselves 
capable, while they evaluate the families and children as incompetent, all school stakeholders must be educated 
about cyberbullying. Nonetheless, the results obtained from our research show that there are very few cyberbullying 
educational activities, especially for families and students, or that such activities are not organized at all. Prevention 
studies related to cyberbullying should be conducted not only at school but also outside school, and geared toward 
the community (Hinduja and Patchin, 2009b). Researchers, on the other hand, mention the importance of peer aid in 
informing students about cyberbullying (Hinduja and Patchin, 2009b; Kowalski et al., 2012). The results of our 
research show that school environment is involved in cyberbullying prevention studies in most schools, while the 
peer-mentoring method is not used. 

Educators’ opinions on the fourth sub dimensions show that the schools’ level of readiness concerning 
taking anti-cyberbullying technological precautions is found to be generally high. To prevent students’ access to 
specific websites or software as part of the cyberbullying prevention efforts, additional software could be installed 
on computers at schools (Hinduja and Patchin, 2009b). Our results show that most of the schools had taken 
hardware/ software precautions on the internet, network and on their computers. 


Conclusion and Recommendations 


It can be concluded that most of the schools in Turkey are ready to address cyberbullying, to the extent of 
creating an anti-cyberbullying school culture and taking technological precautions; however, especially from the 
creating anti-cyberbullying curriculum and education and creating anti-cyberbullying policies standpoint, there is a 
lack of information and study. 

Researchers should concentrate on conducting comparative studies that gathered students, parents, and 
educators. There is also need for studies on awareness of recognition and intervention of cyb erbullying cases, 
together with improving this awareness with educational interventions. School themselves could also engage in self- 
evaluation studies on overcoming cyberbullying and institute policies to improve their anti-cyberbullying standing. 
Schools can start custom programs for their schools working together with universities and experts in the field. 
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Abstract 


This mixed-methods study examines the impacts of a semester-long technology professional development 
for international teachers from developing nations and is a follow up to our previous work in this area. Using the 
TPACK framework and a survey approach, first we investigated the international teachers’ perceptions regarding 
their technology integration abilities. Then, a lesson planning design task was used to qualitatively analyze their 
rationale behind the technology integration decisions and to triangulate. Overall, the results confirmed our earlier 
findings that despite reporting highest growth in the technological content knowledge on the post-survey, 
participants intended to use technology with general pedagogical strategies rather than content specific ones. 
However, the technology access issues were less evident compared to our earlier study, possibly suggesting that 
availability of technology and its adoption in education may be improving in the developing nations. 


Introduction 


Globalization trends, first visible in commerce and industry, have also impacted the field of education. As a 
result, teacher education has begun to embrace more international collaborations and exchange programs in recent 
years (Charney, 2009; Townsend, 2011). One such exchange effort is the International Leaders in Education 
Program (ILEP), conducted by the International Research & Exchanges Board, a non-profit organization established 
with the help of U.S. Department of State’s Bureau of Educational and Cultural Affairs (U.S. Department of State, 
2017). Since 2007, the ILEP initiative has brought exemplary secondary school teachers from developing nations to 
the United States for a semester long program at four host universities. The program encompasses: a) coursework on 
teaching methodologies, curriculum development, and educational technologies, b) field experiences in local high 
schools, c) exposure to civic and cultural activities, and d) participants designing a professional development module 
to enhance teaching and learning in their home-schools. 

A key component of ILEP is the technology professional development course, which aims to develop 
teachers’ ability to integrate technology meaningfully in their teaching. The course is designed to equip the 
international teacher participants with education technology resources, skills, and experiences that they can adapt for 
use in their home classrooms. While there is a plethora of research on technology-related professional development 
of in-service teachers in developed countries such as Australia, United States, Singapore, and Taiwan (Harris & 
Hofer, 2011; Koh & Chai, 2014; Liu, Tsai, & Huang, 2015), limited research has examined technology integration 
expertise of teachers from developing nations (Kozma, McGhee, Quellmalz, & Zalles, 2004; Voogt & Plomp, 2010). 
This research points to two primary barriers that hinder use of technology in education in developing nations. The 
first barrier is the poor technology infrastructure and access issues, and the second barrier is the shortage of skilled 
teachers and trainers who can integrate technology meaningfully in education (Kozma et al., 2004; Makgato, 2012; 
Olakulehin, 2007). Our earlier study (Dalal, Archambault & Shelton, 2017) conducted with the 2016 ILEP cohort 
suggested that though the international teachers learned to consider the affordances of technology in their 
instructional planning after the coursework, technology access issues were pivotal in their choice and use of 
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technology. The current study is a direct follow-up and extension of our previous work to see if we find similar 
results. 
Thus, using a mixed-methods approach, we address the following research questions: 


1. What is the perceived ability of integrating technology in instruction for the international teachers 
before and after a semester long technology professional development course? 
2, How do international teachers approach technology integration in their instruction planning following 


the technology professional development? 
Theoretical Framework 


Lee Shulman (1986) introduced the concept of pedagogical content knowledge (PCK), recognizing the 
need for a more coherent theoretical framework with regard to what teachers should know and be able to do. Within 
PCK, he included “the most powerful analogies, illustrations, examples, explanations, and demonstrations” in a way 
that facilitate powerful teaching and student learning of complex subject matter (p. 9). Koehler and Mishra (2005) 
extended PCK, developing a framework combining the relationships between content knowledge (CK), pedagogical 
knowledge (PK), and technological knowledge (TK) calling it technological pedagogical content knowledge 
(TPACK). TPACK represents the intersection of all three knowledge domains (content, pedagogy, and technology) 
and involves an understanding of the complexity of relationships among students, teachers, content, technologies, 
and practices. As Niess (2005) wrote, “TPCK, however, is the integration of the development of knowledge of 
integration of the different domains that supports teachers in teaching their subject matter with technology” (p. 510). 

Since its formal inception and establishment as a theoretical concept, TPACK has been used by researchers 
to assess technology-related professional development efforts of teachers (Graham, Borup, & Smith, 2012; Altun & 
Akyildiz, 2017; Liu et al., 2015). In examining the effective outcome of a technology-related teacher professional 
development program, TPACK addresses each of the three major components (CK, PK, and TK) and their 
combinations (PCK, TCK, TPK, and TPACK) needed to ensure high quality instruction. Using this framework, the 
current study looks for the evidence of TPACK growth among international teachers following the ILEP technology 
course and also examines the instructional planning approaches of international teachers in relation to TRACK 
constructs. 


Methodology 
Context 


The study took place over a semester long, technology course at a large university in the United States, one 
of the designated host centers of the ILEP program since 2013. The course was divided into 10 sessions consisting 
of three hours each, held from mid-January to April, 2017. The technology professional development was designed 
based on the International Society for Technology in Education (ISTE) Standards for Teachers (ISTE, 2014) and 
focused on methods of effectively integrating computer-based technology in teaching and learning. As part of the 
program, all participants were provided with laptops. The classroom was equipped with access to the internet and 
one projector. 

The technology course was divided into two phases: a) a learning phase with formative assessments and b) 
a transfer phase with a culminating design task. At the start of the course, all participants were given a brief 
overview of the TPACK framework emphasizing that knowing how to use technology hardware and software alone 
does not guarantee effective integration of technology in classroom teaching. Meaningful technology integration 
requires systematic understanding of technology, subject matter or content, pedagogy, and how these domains work 
together (Koehler & Mishra, 2005). 

The learning phase involved face to face class instruction with hands-on tutorials on topics such as learning 
theory and technology, digital citizenship, internet safety, web-based communication tools, student and teacher 
productivity tools, video recording and editing, screencasting, web-based assessment applications, and mobile/online 
learning. The hands-on tasks varied in complexity from simple synthesis of information such as understanding 
online safety issues to complex implementation of ideas like creating digital stories or Google forms. At the end of 
each session, participants were asked to create an artifact either individually or in a small group to demonstrate their 
understanding and comfort level with the technology (e.g., create a slideshow, participate in online discussion group, 
or produce a digital story). These artifacts and their presentations served as formative performance assessments for 
the course. 
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The transfer phase involved a culminating design task. In the last two weeks of the course, all participants 
were asked to develop one lesson that integrated at least one educational technology tool they had learned. The 
design task not only demonstrated the ability of the teacher participants to conceptualize, design, and deliver a unit 
plan infused with technology, but it also provided an opportunity for the teachers to engage in decision making, 
while practicing technology integration in their own curricula. 


Participants 


The study involved 16 international, secondary school teachers, divided equally between males and 
females, who participated in the Spring, 2017 ILEP cohort. Their countries of origin were Bangladesh, Brazil, 
Ghana, India, Indonesia, Malaysia, Morocco, and Tanzania. The average age of participants was 38 years, ranging 
from 24 to 49 years. Their teaching experience ranged from six to 21 years, with an average of 12 years. Participants 
reported a wide range (four to 23 years) of exposure and experience with computer technology (Table 1). 


Table 1. Profile of Participants 


Professional and technology experience Response count Response % 


Content area 


Language Arts 10 62 
Science 3 19 
Social Studies 3 19 
Started using technology 
Prior to Age 10 0 0.0 
Age 10 to 15 2 12 
Age 16 to 20 4 25 
Age after 20 10 63 
Exposure to computers 
1 to 5 years 3 19 
6 to 10 years 5 31 
11 to 15 years 4 25 
16 to 20 years 3 19 
21 to 25 years 1 6 
Time spent on computer in a typical work-day 
No time 0 0 
Less than one hour 5 31 
About 1| - 2 hours 4 25 
About 3 - 4 hours 5 31 
5 or more hours 2 13 
Comfort level using computer 
Not at all comfortable 0 0 
Somewhat comfortable 5 31 
Comfortable 7 44 
Very comfortable 4 25 


Data Sources 


Survey. To answer the first research question regarding the perceived ability of international teachers to 
integrate technology in instruction, we used a survey instrument created previously (Dalal et al., 2017). The 
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instrument was based on the validated TPACK measure (Archambault & Crippen, 2009), but modified for an 
international teacher population. The goal was to measure teachers’ confidence related to all TPACK constructs 
using closed items on Likert scale. The online survey was administered both before and after the course. 

Design task. Over a two-week period at the culmination of the course, all participants were asked to design 
one lesson plan using at least one educational technology tool in the instruction. The participants were able to 
choose the technology based on its affordance and the subject matter they wanted to address. Along with the lesson 
plan, participants submitted a written rationale for their design decisions. This rationale described why they chose a 
particular technology as well as how the technology would be used by the teacher and/or students. All participants 
submitted lesson plans and rationale statements online using the course management system. 


Limitations 


It is important to acknowledge the limitations of this study. A survey instrument is inherently limited by its 
items and scales (Fowler, 2002), and self-report measures are susceptible to bias (Spector, 1994). Additionally, the 
study’s sample size was small and represents a limited population of international teachers, thus restricting the 
generalizability of the findings. Moreover, a different design task could yield different results for understanding the 
technology integration approaches of international teachers. 


Analysis and Results 
Survey: Perceived Ability of Technology Integration 
The Likert-scale TPACK items were assigned values as follows: not at all able (1), rarely able (2), 
sometimes able (3), often able (4), and most of the time able (5). For each individual item, we calculated descriptive 


statistics both pre and post coursework (Table 2). Then, descriptive statistics were calculated for the seven TPACK 
constructs and internal reliability was established by verifying Cronbach’s alpha (Table 3). 


Table 2. Descriptive statistics for all individual items 


Pre Post 
Construct Individual items M SD M SD 
TK I can solve technical problems associated with my computer or 2.94 0.93 3.69 0.95 
network connection. 
I can address various computer issues related to software (e.g., 2.62 0.81 4.00 0.97 


downloading appropriate plug-ins, installing programs). 
I can assist students with solving technical problems with their 2.25 0.86 3.56 0.89 
personal computers. 


PK I can use a variety of teaching strategies to share different 3.25 0.86 4.75 0.58 
concepts with students. 
I can modify instruction based on my assessment of students’ 3.69 1.19 4.56 0.73 
knowledge and skills. 
I can determine a particular teaching or learning strategy best 3.37 1.15 4.60 0.63 
suited for a particular concept. 
I can adjust my teaching methodology based on student 3.19 1.33 4.56 0.73 


performance or feedback. 


CK I can create materials that align with the concepts students need 3.62 0.96 4.44 0.73 
to learn in a course. 
I can decide on the concepts in my content area that should be 3.62 1.15 4.56 0.73 
taught in a class. 


I can plan the sequence of concepts taught within my class. 3.5 1.04 4.53 0.64 
I can produce written lesson plans for the classes I teach. 4.19 0.91 4.60 0.63 
TPK I can create a technology-rich learning environment where 2.56 1.15 4.38 0.88 


students can build new knowledge and skills. 
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I can implement different methods of teaching with 2.56 0.89 4.33 0.72 


technology. 
I can use technology to facilitate communication and 3.00 1.32 4.63 0.50 
interactions among students. 
I can motivate students to want to use technology for 3.56 1.09 4.80 0.41 
communication and interactions with each other. 

PCK I can distinguish between correct and incorrect problem 3.50 1.09 4.47 0.74 
solving attempts by students. 
I can anticipate likely student misconceptions regarding a 3.31 1.14 4.31 0.79 
particular concept. 
I can assist students in noticing connections between various 2.56 1.26 4.38 0.72 


concepts in a curriculum. 


TCK I can use technological representations (like an online video, an 3.06 1.06 4.47 0.74 
interactive map, or an online science experiment simulation) to 
demonstrate particular concepts in my content area. 
I can implement curriculum using technology. 2.00 1.15 4.47 0.64 
I can use various online courseware programs to deliver 2.50 1.17 4.31 0.79 
instruction (e.g., Edmodo). 


TPACK I can use technology to predict students’ skill/understanding of 2.19 1.11 4.19 0.83 


a concept. 

I can use technology to create effective representations of 2.94 1.00 4.50 0.73 
content that expand on the content presented in my textbook. 

I am able to meet the overall demands of teaching a class, 2.81 1.17 4.40 0.83 


using technology. 


Table 3. Descriptive Statistics for TPACK Constructs and Reliability Scores 


Pre-survey Results Post-survey Results Post-Pre Cronbach’s Alpha 

Construct M SD M SD M 

TK 2.60 0.87 3.75 0.94 1.15 .691 
PK 3.37 1.13 4.62 0.67 1.25 751 
CK 3.73 1.01 4.53 0.68 0.80 627 
TPK 2.92 1.11 4,53 0.63 1.61 .788 
PCK 3.12 1.16 4.39 0.75 1.27 .763 
TCK 2.52 1.13 4.42 0.72 1.90 817 
TPACK 2.65 1.09 4.36 0.80 1.71 821 


Note. Means and standard deviations computed from responses (N = 16) on the following scale: not at all able (1), rarely able (2), 
sometimes able (3), often able (4), and most of the time able (5). 


Design Task: Technology Integration in Instructional Planning 


To address the second research question regarding international teachers’ approach to technology 
integration in their instructional planning, the lesson plans (N=16) and accompanied rationale statements were 
analyzed qualitatively. We used a-priori TPACK codes in combination with open coding to understand the 
technology-related decision making of the international teachers (Corbin & Strauss, 2015; Tracy, 2013). We decided 
to use both inductive and deductive coding due to limitations of TPACK framework. While the framework is 
theoretically robust and helpful for organization, the fuzziness of the domains make it difficult to separate out each 
of the domains, limiting its practical use (Archambault & Barnett, 2010; Kimmons, 2015). In addition, given the 
new themes beyond the TPACK codes identified in our previous work, including affordance of technology, student- 
centered learning, discovering new technology, access issues, and teacher-centered learning (Dalal et al., 2017), we 
wanted to look for similar evidence, if present. 
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From the submitted design task materials, we identified 94 statements as data or meaning units for further 
content analysis (Merriam, 1998). Statements were identified as meaning units for analysis if they pertained to either 
a specific educational technology, a justification for choosing that technology, or its intended use. These statements 
were extracted and saved in a spreadsheet for further analysis. Together, two members of the research team read and 
re-read the data units, first coding for one of the presence of the four technology-related TPACK domains (i.e., TK, 
TPK, TCK, or TPACK). In a second round, open coding (Corbin & Strauss, 2015) was used on the same meaning 
unit to identify potential additional themes. The data units were repeatedly read and codes were consolidated or 
expanded as necessary, until coding saturation was reached. Table 4 outlines the 11 codes thus identified with their 
description and a sample example. Then one researcher coded the remaining data units (78%) which were reviewed 
by the other members of the team. Any discrepancy was resolved by discussion to reach 100% agreement. 


Table 4. Codebook with examples 


Code Description % Example 
Evidence 
TK.af Teacher describing what the 11 “Google Forms allows you to ask both 
(affordance) technology adds to their lesson plan open-ended and closed-ended questions.” 
considering the benefits or advantages 
of technology in education. 
TK.re Teacher thinking about available 5 “T chose Powerpoint and Youtube because 
(resource technology resources and school these are the only resources that can be 
constraints) policy constraints. used in my school.” 
TPK.is Use of technology for a specific 21 “After viewing a YouTube video, 
(instructional instructional strategy or a particular encourage online discussion by asking 
strategy) pedagogical goal such as assessment. learners to add their personal insights: 
What did they like about the video? Was 
there anything they didn’t understand? 
How did the video relate to their personal 
experiences/feelings?” 
TPK.af An instructional strategy that involves 75 “Using YouTube videos in lesson 
(affordance) use of technology specifically encourages your audience to develop their 
considering the affordance of the note-taking skills by viewing, rewinding, 
particular technology tool. and replaying the video until they have 
fully grasped its essence and key points.” 
TPK.lu An instructional strategy that involves 75 “As a teacher of classes with inclusion 
(learner use of technology but keeping in mind students and at risk students, I have found 
understanding) the background, abilities and that measurable differences occur in 


motivations of the learner. 
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learning and retention ... Algebra tiles 
website can assist their learning process 
from concrete to symbolic.” (learner 
background) 


“Skills such as taking notes, writing 
summary, or reading comprehension are 
being taught through interesting 
approaches such as taking note from 
videos and online articles instead of 
reading textbook and listening to the 
teacher.” (engagement and motivation) 


“Students with hearing loss learn by 
watching and observing, so with 
technology, they will see the presentation, 
my delivery using signing language, and 
observe the graphic organizer 


TCK.af 
(affordance) 


TCK.re 
(resource 
constraints) 


TCK.tp 
(teacher 
planning) 


TPACK.is 
(instructional 
strategy) 


TPACK.af 
(affordance) 


TPACK.lu 
(learner 
understanding) 


Teacher talking about a particular 
technology tool considering its 
affordance and relevance for specific 
content. 


Considering a technology tool for its 
relevance to specific content while 
weighing the technology access issues 
and resource constraints. 


Teacher planning the lesson thinking 
about content and technology. 


Teacher combining pedagogy, content 
and technology in instructional 
strategy however, one domain may be 
prevalent. 


An instructional strategy for a 
particular content that involves 
meaningful use of technology 
considering the affordance of the 
particular technology.” 


Teacher talking about an instructional 
strategy for a particular content that 
involves use of technology but 
keeping in mind the background and 
abilities of the learner. 


2.5 


25 


representing the processes of debate.” 
(learner ability) 


“Chemistry experiments are hazardous, 
one cannot see what’s happening at 
molecular level. PhET allows to see and 
conduct experiments again and again if 
something goes wrong.” 


“Show google earth as it is useful for 
geography lesson. No internet in school 
but has personal connection through 
mobile. The plan to show google earth 
may fail.” 


“Power point will be used in early stage, 
(New knowledge presentation) to explain 
the content of the lesson.” 


“The teacher introduces VoiceThread, an 
online platform where students can leave 
their comments on a question/subject 
given by the teacher. This kind of activity 
can help students improve their listening 
and speaking skills.” (Pedagogy) 


“Students will watch the video about 
exploring the coral reef. Students will 
discuss and pose questions about the 
video they have watched.” (Content) 


“They will learn how to produce their 
learning products by making Prezi 
presentation, power point show, movie 
maker and PhotoPeach.” (Technology) 


“Stop the video at 4:59 minutes and solve 
the following sums from your textbook 
based on arithmetic mean. You may 
watch the video again for clarity and 
understanding.” 


“Students often have difficulty learning 
Mathematics especially algebra. It is 
deemed too abstract and students cannot 
represent the numbers and variables 
physically. But students can understand 
algebraic thinking and its concepts with 
Algebra tiles website that takes geometric 
approach to algebraic concepts.” 


Note. Percentages calculated based on N = 94 rationale statements. 


Overall, the largest portion (40%) of the data units were related to TPK followed by TK (18%), TPACK 
(15%), and TCK (8%). Of the statements, 4% could not be coded using a second level as they did not reflect a 
particular or consistent theme. Thus, total 81% of the data units were coded for one of the technology related 
TPACK constructs (TK, TPK, TCK or TPACK). The remaining 19% data units simply mentioned specific 
technologies that international teachers were planning to use in instruction upon return. 

These included statements such as, “Students will need mobile phones with internet connection and 
preinstalled PhEt lab application.” or “This Khan Academy video is perfect to teach basic statistics.” Specific 
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technologies mentioned by teachers included Kahoot, Voice Thread, YouTube, Prezi, PowerPoint, Moviemaker, 
PhotoPeach, Screencast-O-Matic, Google Forms, Audiobook, Poll Everywhere, and the Illuminations website from 
National Council of Teachers of Mathematics. 

Examining the secondary level codes, 30% of the statements were related to instructional strategy, 27% 
suggested thinking related to various affordances of technology, 10% reflected teachers’ understanding of their 
students’ backgrounds and abilities, 8% conveyed technology access and resource issues, and 2% of the statements 
indicated teacher planning. Remaining statements were related to either specific technology names, websites or the 
4% of statements that could not be coded for a secondary level. 


Discussion and Implications 


ILEP participants form a highly heterogeneous group coming from a variety of nations, teaching different 
subjects and grade levels, and possessing varying levels of technological experience. As was the case with our 
earlier study (Dalal et al., 2017), the participants began the ILEP professional development program with greater 
levels of confidence in their CK (3.73), PK (3.37), and PCK (3.12) but lower knowledge levels in all technology- 
related constructs. This was expected with the average teaching experience of 12 years among the participants, 
resulting in higher confidence for content knowledge and teaching strategies. However, after completing the ILEP 
program, participant confidence levels for technology-related constructs advanced significantly to match with other 
non-technology related construct values; and the biggest gains were reported in technology-related constructs of 
TCK (1.90), TPK (1.61) and TPACK (1.71). This evidence suggests that the ILEP program’s emphasis on 
educational technology was an effective way to empower the international teachers as confident, more assured 
integrators of technology. 

The qualitative analysis of the lesson plans and rationale statements, however, suggests greater 
development of TPK compared to TK, TCK, or TPACK. The international teachers included in the study were 
highly experienced. In addition, the presence of categories of learner understanding such as TPK.lu and TPACK.lu 
suggest that these teachers were confident that they knew their students well, which again speaks to their experience. 
Our approach, which leveraged teachers’ existing pedagogical skills to impart new technological knowledge may 
not be as effective for international teachers with less teaching experience. This is consistent with Graham et al. 
(2012) assertion that building a strong foundation of PCK is key for developing TPACK. 

While participants reported the highest confidence gain in TCK (1.90), the design task analysis showed 
very little evidence (8%) of TCK. It seems that while teachers felt confident in integrating technology across their 
content areas, their ability to do so in the lesson planning process was still limited. This finding matches the results 
of our earlier study (Dalal et al., 2017). One consideration is that technology access issues in the developing nations 
(Buabeng-Andoh, 2012; Voogt & Plomp, 2010) could impact how these teachers designed their lessons. Six out of 
16 teachers clearly indicated lack of internet or computers in their school. They were also aware that not all students 
have access to computers and internet at home. These technology resource constraints would act as gatekeepers, 
preventing them from designing lessons suggestive of TPACK where technology is used constructively with full 
involvement of students (Koehler & Mishra, 2005). However, the technology access issues and resource constraints 
were less noticeable in only 8% of the meaning units as compared to the previous ILEP study (Dalal et al., 2017) 
where we reported 14% meaning units reflecting technology resource issues with the same number of participants 
(n=16). This could be an indication of increasingly improving technology infrastructure and penetration of mobile 
technology in the education scene of many developing nations (Chaudhuri, 2012; UNESCO, 2015). 

Other considerations for lower evidence of TCK in lesson plans could be that while there is an explosion of 
educational technologies for assessments and content-sharing, the technologies themselves are rarely designed with 
teaching and learning specific content in mind (Laurillard, 2009; Forssell, 2016). Acknowledging this limitation. 
Forssell (2016) implored designers to leverage teacher knowledge and create digital tools guided by the TPACK 
framework. Moreover, researchers (Graham et al., 2009; McCrory, 2008) have indicated that during technology 
professional development it is easier to integrate technologies for general teaching strategies than the technologies 
intended for specific content. This was a particular challenge in the ILEP technology course because the teacher 
participants had varied backgrounds in terms of content area, grades, and also curriculum owing to different 
nationalities. The higher concentration of data units coded for instructional strategy (30%) suggest that while 
designing lessons teachers were thinking about pedagogy — how were they going to teach a particular topic, what 
difficulties their students might face, and how were they going to assess the learning. As Archambault and Barnett 
(2010) specify, the teachers are inherently not thinking about technology skills alone or separating out domains of 
content, pedagogy, and technology when planning. They are likely considering these elements together, all at once. 
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An indication of the success of the ILEP technology course was evident when the theme of affordance was 
noticed in the second level coding of all technology-related constructs including TK.af, TPK.af, TCK.af, and 
TPACK.af. The knowledge of affordance is suggestive of advanced thinking about how technology can enhance 
content and pedagogy (Koehler, Mishra, Kereluik, Shin, & Graham, 2014; McCrory, 2008). The teachers learned to 
consider the benefits and advantages of technology at every stage of instruction planning to see if the teaching- 
learning process could be enhanced by the use of specific technology tools, rather than simply using the technology 
for the sake of doing so. 

In summary, the international teachers entered the ILEP program with limited confidence in their ability to 
integrate technology in the classroom. During the semester-long technology course, they explored new educational 
technologies, built upon their integration skills, and learned to think about the affordances of these technologies in 
connection with their instructional strategies. Through other activities in the ILEP program, such as observing in 
local high schools and taking other university level courses, they also had varied experiences with educational 
technology in action. As was the case with our earlier study (Dalal et al., 2017), the results suggest that while 
teachers learned to consider the affordances of technology in instruction planning, there remains a distinct need to 
impart general technology skills and awareness of content specific educational technology tools. 

Future work along this line would benefit from similar studies with higher number of participants or with a 
different design task to explore the decision-making approaches of international teachers. A longitudinal study 
examining the actual classroom practices would illuminate not only the ability of international teachers to integrate 
technology in the natural settings of their home countries, but would also shed light on the new approaches needed 
in the design of technology-related professional development programs for international teachers. 

The findings have implications for the future design and motivation of international teacher education 
programs. International collaborations and teacher exchange programs such as ILEP are the vehicles to provide 
space and opportunity for teachers to make connections between and among diverse educational issues and engage 
in meaningful learning that enables them to use technology constructively. Lawless and Pellegrino (2007) have 
emphasized that the digital divide between urban and rural schools in the United States is not because of a lack of 
access to technology, but due to the lack of access to teachers who are able to integrate classroom technology 
effectively. This discovery has even larger implications in the context of developing nations where there is a critical 
and continuing shortage of teachers and trainers equipped to integrate technology in education (Kozma et al., 2004; 
Makgato, 2012; Olakulehin, 2007). We anticipate that the current study will provide added focus on the technology- 
related professional development needs of developing nations and contribute toward longitudinal research in 
understanding the sustainability of technology professional development efforts in the international context. 
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Introduction 


Self-assessment, which is a cyclic process of self-monitoring, self-evaluation, and identification and 
implementation of instructional correctives as needed, is fundamental and critical skill for lifelong learners (Yan & 
Brown, 2016). Lifelong learner needs to frequently assess individual performance, validity of learning resources, 
and the learning environment. Self-assessment affects learner’s cognition, affection, and conation and assists 
learners to learn how to learn. Effective self-assessment can encourage deep as opposed to surface approaches to 
learning, with positive implications for the quality and sustainability of learning (McDonald & Boud, 2003). 

Formative analytics has been identified as a good method to support students’ self-assessment competency 
development (Singh & Terry, 2008; Zhong, 2016). As a new approach of learning analytics, formative analytics 
focus on supporting the learner to reflect on what has been learned, what can be improved, which goals can be 
achieve, and how to move forward. By providing analytics for learning rather than analytics of learning, formative 
analytics can empower each individual learner to reach his or her potential through personalized feedback and self- 
reflection 

Though many theoretical discussions related to self-assessment and learning analytics have been 
conducted, empirical studies exploring how self-assessment competency can be supported through formative 
analytics appear to be scarce. Aimed at filling this research gap and get deeper understanding of how formative 
analytics can support self-assessment competency development, this study examined the effects of formative 
analytics on developing self-assessment competency in workforce education. The research question, How formative 
analytics impacts students’ self-assessment competency development?, were examined in this study. 


Literature Review 


Self-assessment is defined as a cyclic process of self-monitoring, self-evaluation, and identification and 
implementation of instructional correctives as needed (McMillan & Hearn, 2008). In this study, self-assessment is 
considered as a learning strategy rather than a personal skill or a type of assessment method (Yan & Brown, 2016). 
Self-assessment has been recognized as fundamental and critical skill to lifelong learners (McDonald & Boud, 
2003). Self-assessment affects learner’s cognition, affection, and conation and assists learners to learn how to learn, 
which is an important lifelong learning competency in digital age. Effective self-assessment should neither 
overestimate nor underestimate learners’ performance. Boud and McDonald (2003) believed that effective self- 
assessment can encourage deep as opposed to surface approaches to learning, with positive implications for the 
quality and sustainability of learning. 

Self-assessment practices cover two major actions, self-directed feedback seeking and self-reflection (Yan 
& Brown, 2016). Self-directed feedback seeking refers to the process by which students initiate and take 
responsibility for seeking feedback from various sources for the purpose of self-assessment. ‘Self-directed’ is used 
to emphasize that the characteristics (the content, source, direction, etc.) of feedback are determined by the student 
him/herself rather than following external instructions. Feedback could be obtained from both external and internal 
sources (Butler & Winne, 1995). External feedback can be obtained from two sources including people and 
processes (Sargeant, Mann, van der Vleuten, & Metsemakers, 2008), or inquiry and monitoring (Ashford & 
Cummings, 1983). Internal feedback comes from within the self, such as emotions, motives, physical sensation and 
internal states (Epstein, Siegel, & Silberman, 2008). Self-reflection is the action by which students reflect on and 
evaluate the quality of their learning process and outcomes with the support of available/gathered feedback, and 
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identify their own strengths and weaknesses. By critical reflection, students may have a better understanding in 
terms of their strengths and weaknesses as well as how to facilitate achieving their goals. 

Adaptive digital learning environment with low pressure is suggested to help learner’s development of self- 
assessment competency ( Yan & Brown, 2016; Zhong, 2015). Students need opportunities to expose their reflection 
without fear of shame or embarrassment. Adaptive learning environment can provide space and time for students’ 
deep thinking. Instructors are suggested to create such learning environment where students are allowed not to 
disclose their self-assessments to the teacher or classmates, but are encouraged to share these with trusted people 
(Andrade & Brown, 2016). 

Formative analytics has been identified as a good method to create adaptive learning environment that 
supports students’ self-assessment competency development (Zhong, 2016). Formative analytics refers to using 
analytics to encourage students to reflect on how they are learning, or to initiate a conversation between a tutor and a 
student, instead of using the system to assess the student’s performance or ability. As a new approach of learning 
analytics, formative analytics focus on supporting the learner to reflect on what has been learned, what can be 
improved, which goals can be achieve, and how to move forward. By providing analytics for learning rather than 
analytics of learning, formative analytics aim to empower each individual learner to reach his or her potential 
through personalized feedback and self-reflection. 


Method 


Case study was chosen to answer the research question. Participants were 8 students in workforce 
education at a large research university in Illinois. All participants are adult learners and have full-time jobs. 
Average age of participants is 34 years old. Data was collected from individual interviews and group interview with 
a semi-structured questions guide. In addition, students’ academic outcomes, in this study ePortfolios (Zhong & 
Hartsell, 2015) and presentations, were included in data analysis. In this course, students were required to complete 
four group projects and three professional presentations and post a weekly reflection on the learning experience of 
the past week on their individual e-portfolios. Interviews and students’ academic outcomes were transcribed and 
coded after students’ grades were submitted. 


Findings 


Preliminary findings showed that formative analytics is effective in supporting self-directed feedback 
seeking. More student-initiated feedback seeking behaviors were reported. It indicates that formative analytics has 
positive impact on self-directed feedback seeking. Students was triggered by formative analytics results to 
independently assess their performance by using the same criterions as the instructor. In addition, internal self- 
directed feedback seeking behaviors were also found from students’ interviews. Students tend to regularly reflect on 
performance not only in this particular course but also in other courses and even personal activities out of school. It 
shows that formative analytics not only affects students’ academic performance but also performance out of schools. 
This finding shows the potential of formative analytics in developing students as lifelong learners, who are 
independent and self-regulated learners. 

Unfortunately, self-reflection behaviors were not found in this study. All reflections behaviors performed in 
the class were required by the instructor. For example, students are required to post a weekly reflection to their 
ePortfolios. All students did what the instructor asked but no additional posts. Although some internal self-reflection 
activities (e.g., thinking in head) were mentioned in students’ interviews, few students went back and checked their 
thoughts again after the class. Instructors still lead self-reflection processes and activities as found from previous 
research (Costa & Kallick, 2004; Zhong, 2017). One possible explanation may be students’ age. Because all students 
are adult learners and have full-time jobs, students are not able to spend extra time on school work. Most students 
tend to meet the minimum requirements of the class in order to get the course completed. Thus, even though 
students had some self-reflections in head, no behaviors were observed. Another possible reason is the technical 
limitation in this study. Although there are some learning data related to self-reflection generated such as website 
tracking data, few of the data can be used to analyze students’ self-reflection activities. In addition, as mentioned 
above, most self-reflection activities were occurred in students’ heads. It is difficult to observe and capture. 
Therefore, researchers are not able to conclude whether formative analytics impacts self-reflection activities. 
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Conclusion 


This qualitative study examined students’ self-assessment competency development supported by 
formative analytics. Preliminary findings showed that formative analytics is effective in supporting self-directed 
feedback seeking but self-reflection was not found although students showed positive attitudes during interviews. 
This study demonstrated that formative analytics has the potential to achieve the goal of developing students as 
lifelong learners. Although few self-reflection behaviors were found in this study, I believe it is possible to promote 
self-reflection behaviors if formative analytics are properly designed and integrated into instruction. Thus, following 
step of this study is to improve formative analytics design in this course to ensure more data related to self-reflection 
can be captured and analyzed. 
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Introduction 


The issue of student identification and authentication is now an essential area of compliance within federal 
policy and law. Providers of online education must develop policies and procedures for verifying that the same 
student who signed up for the course does the work and receives the academic credit. Protecting the integrity of 
online courses and programs, and satisfying the accrediting agencies responsible for enforcing the law, requires 
investment of time and resources into the prevention and detection of academic dishonesty. 


Federal Compliance 


In August 2008, Congress passed the Higher Education Opportunity Act, reauthorizing the 1965 Higher 
Education Act, as amended. In a section addressing accreditation and program integrity, 34 C.F.R. § 602.17(g), the 
law states in pertinent part: 

[The agency] Requires institutions that offer distance education...to have processes in place through which 

the institution establishes that the student who registers in a distance education ...course or program is the 

same student who participates in and completes the course or program and receives the academic credit.... 

The statute specifically requires institutions to verify the identity of a student who participates in a class or 
coursework by using methods such as: secure login and passcode; proctored examinations; and, new or other 
technologies and practices that are effective in verifying student identity.... 34 C.F.R. § 602.17 (g)(1)@ -111). 


Ways to Ensure Academic Integrity 
Educators in higher education use a variety of methods and tools to ensure academic integrity in online 
courses. Creating a clear academic dishonesty policy and making them availability to students is the first step 


(Simonson, Smaldio, Albright, & Zvacek, 2003). Utilizing authentic assessment strategies, such as collaborative 
projects, and e-portfolios is effective (Bobak, Cassarino, & Finley, 2005). Requiring an outline and a draft for 
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individual written projects prevents students from submitting others’ work in the end of a semester. Wikis are a great 
tool in the implementation of this process. Many software applications like Turnitin and SafeAssign can also detect 
plagiarism by comparing students’ essays to electronic databases. 

Proctored examinations are another way to verify student identification. Whether it is prepackaged, using 
lockdown browser and recorded video features in the Learning Management System (LMS), or proctored with a 
human proctor at a distance via video streaming, consideration must be given to convenience, affordability, minimal 
hardware requirements, and most importantly, security. 

Prince, Fulton, and Garsombke (2009) found significant differences in average test scores between tests 
taken electronically without a proctor and those administered using a live or a remote proctor. Students scored 
significantly lower on proctored exams versus non-proctored exams. Cochran, Troboy, and Cole (2010) found that 
the grades for the remote proctor group trended lower than the non-participants. They also found that remote proctor 
participants felt that the remote proctoring system had no impact on their exam taking abilities. 

To meet the need of online education in online testing, many vendors have developed technological tools to 
provide proctoring services. This study involved Respondus Monitor, Respondus LockDown Browser, and 
Blackboard Test Tools available to faculty and students in the university that the study was conducted. 


Non-proctored Recorded Online Testing Environment 


Respondus Monitor is a companion product for LockDown Browser that enables students to take online 
exams in non-proctored environments (“Respondus Monitor, 2015”). There is no additional software to install, and 
students use their own computers with a standard webcam to record assessment sessions. Instructors can review the 
entire session from Blackboard course sites. If used, students enrolled in the course need to pay a flat fee per course 
per semester. Institutions can also purchase seats and allow students to use it for free. 


Non-proctored Lockdown Online Testing Environment 


Respondus LockDown browser is a custom browser that locks down the testing environment and can be 
integrated with the LMS. When students are required to use the LockDown Browser when taking a test, they are 
unable to print, copy, go to another website, or access other applications using the same computer. When a test is 
started, students are locked into it until they submit it for grading (“LockDown Broswer”, 2015). 


Non-proctored Online Testing Environment 


The Blackboard quiz/survey tool allows faculty to create timed tests with random selections pulled from a 
pool of questions. On the test day students will be able to take the test in the time frame that was set by the faculty. 
The questions in the test can be different from student to student since they can be randomly selected from the 
question pool. When time expires, the system will shut down the test. 

The Blackboard quiz tool allows faculty to control when the results of the test are released to students, as 
well as the type of results and feedback that are released. Instructors are encouraged to consult with an instructional 
designer to discuss options and settings that will meet the goals of individual assessments, and the course as a whole. 


The Study 


The purpose of this study was to determine which remote testing systems, all designed to ensure academic 
integrity, offer a realistic and secure approach to online examinations and best fulfill the needs for faculty and 
students. This study explored the use of three testing tools to determine whether there are differences in test scores 
and student grades in the distinct testing environments. They are: Non-proctored Recorded Online Testing 
Environment (Respondus Monitor), Non-proctored Lockdown Online Testing Environment (Respondus LockDown 
Browser) and, Non-proctored Online Testing Environment (Blackboard Test and Survey Tool). The data was 
collected from a Midwest public university. 


About the Course in the study 


This course on auditing was a cross-listed, offered to both undergraduate and graduate students, and open 
to all students at the university. The same faculty member has offered this course online for several semesters. The 
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overall goal of the course was to help prepare students for a professional career either working as an auditor or 
working with auditors. The course introduced numerous professional topics such as the importance of auditing, 
management assertions, risk, evidence, and reporting and professional liability. The material covered is tested in a 
standalone section on the certified public accounting (CPA) professional exam. 

The course included two midterm exams and one comprehensive final exam. The exams contained true or 
false and multiple-choice questions, several cases, and a four-part company analysis project. The final exam was 
weighted. Approximately one half of the points were based on material already tested. The other half was based on 
two chapters that were not previously tested. 

To assure the integrity of the testing process different testing tools were used in the previous semesters. In 
summer and fall 2014 and spring 2015 semesters, Respondus Monitor was used. In spring 2014 Respondus 
Lockdown was used. In summer and fall 2013 Blackboard Test tool was used. The settings for all exams were set 
up the same with randomized questions, displayed one at a time, and backtracking prohibited. The exams opened 
from Sunday through Wednesday with 80 minutes to complete the final exams and 55 minutes to complete each of 
the two midterm exams. 


Data Collection and Analysis 


The instructor collected the exam scores and final grades of the same course offered over four semesters 
with different testing tools used. The instructor also removed identifiable student information before data analysis. 
Eighty-seven students used Respondus Monitor, 32 students used Respondus LockDown Broswer, and 38 students 
used Blackboard Test Tools. One-way ANOVA was used to determine the grade difference between the three 
groups. Among the 87 students whose tests were administered in Monitor, | student did not take any of the tests; 
therefore, that data set was dropped from analysis. 


Results 


Descriptive statistics were calculated on the mean scores of exams administered in three testing 
environments: Non-proctored Recorded Online Testing Environment, Non-proctored Lockdown Online Testing 
Environment and, Non-proctored Online Testing Environment. As shown in Tables 1-3, there is not much difference 
in the mean score between three different testing environments. However, the standard deviation scores of the final 
exam and the total points between the three testing environments are different and it is very large as there are two 
students who did not take some of exams when the tests were given in Respondus Monitor. 


Table 1. Mean and Standard Deviation of Exams Administered in Monitor 


Exam 1 83.49 12.009 
Exam 2 84.28 15.459 
Final Exam 127.37 25.421 
Total Points : : 389.907 71.1879 
Valid N (listwise) 


fd 
Exam 1 42 98 81.94 14.313 
Exam 2 58 100 83.82 10.409 
Final Exam 86 148 132.18 15.559 
Total Points 336.5 479.5 418.424 41.6126 
Valid N (listwise) 
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Table 3. Mean and Standard Deviation of Exams Administered in Blackboard 


Exam 1 58 96 80.56 10.789 
Exam 2 52 100 84.00 11.211 
Final Exam 84 150 125.18 17.881 
Total Points 269.0 512.0 423.013 56.3328 
Valid N (listwise) 


One-way analysis of variance (ANOVA) was used to determine whether differences existed in the mean 
score of tests administrated in three testing environments. As shown in Tables 4-7, no statistical differences were 
detected across three testing environments in mean scores of the exams. 


Table 4. ANOVA results for Exam 1 by method of testing environments 


Between Groups 21.194 23 921 1.375 135 
Within Groups 89.825 134 .670 
Total 111.019 157 


Table 5. ANOVA results for Exam 2 by method of testing environments 


Between Groups 23.242 23 1.011 1.543 .067 
Within Groups 87.777 134 655 
Total 111.019 157 


Table 6. ANOVA results for Final Exam by method of testing environments 


Between Groups 28.449 29 981 1.521 059 
Within Groups 82.570 128 645 
Total 111.019 157 


Table 7. ANOVA results for Total points by method of testing environments 


Between Groups 84.019 116 .724 1.100 372 
Within Groups 27.000 41 .659 
Total 111.019 157 


Table 8. Percentages of Letter Grades 


Environments/Grades 
A Cc D&F 


Monitor 


Lockdown 


Blackboard 
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Discussion and Implication 


Although there were less students with Ds and Fs when exams were administered in Respondus Monitor 
(4.62%) than in Lockdown Browser (9.12%) and Blackboard Test Tool (5.09%), there was a big difference between 
the three testing environments on the standard deviation of the final exam (Monitor, 25.42; Lockdown, 15.56; 
Blackboard, 17.88) and the total scores of the exams (Monitor, 71.19; Lockdown, 41.61; Blackboard, 56.33). The 
difference might indicate that the technology-based non-proctored testing tool via streaming audio and video, such 
as Monitor, could be the environment that discriminated the student population in this study. 

The result was not statistically significant, but the standard deviations of exams administered in Monitor 
were bigger than exams administered in the other two environments. It is recommended that technology-based non- 
proctored testing tool via streaming audio and video, such as Monitor, be used for high-stake exams if human 
proctored testing is not feasible to students due to a variety of constraints such as cost, time to travel, and schedule. 
However, other types of assessments and online testing can be used but need to be carefully designed and 
implemented in online courses. 

The possibility of academic dishonesty can be better handled by proactively assisting our students develop 
legitimate strategies and providing resources and support throughout their academic journey. When designing 
student assessments, faculty can consider different types of assessments to evaluate student performance against the 
stated learning objectives. The assessments can include but are not limited to: online testing; online proctored 
testing; on campus testing; remote proctored testing; and, authentic assessment. 

No matter what type of assessments faculty use, it is imperative to have an open, ongoing dialogue with 
students about academic dishonesty and the consequences of breaching the policy. Embedded activities regarding 
academic dishonesty scheduled for the first week of class, such as a pop-quiz or an online discussion, may be an 
effective way to communicate the policy to your students. 

This study was based on a course offered in different semesters with limited number of students and was 
not statistically significant. However, if human proctoring is not a feasible option for students, the findings 
indicated that technology-based proctored testing tool via streaming audio and video, such as Monitor, may be an 
effective tool to use to tackle the issues of academic integrity, especially for high-stake tests such as midterm and 
final exams. Further studies are necessary utilizing larger data sets. 
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Abstract 


This paper will analyze the implementation of an active learning strategy built on productive failure in a 
foundational course in a pharmacy doctorate program. This strategy followed several non-graded tasks, aligned with 
effective strategies for failure-based activities. A quasi-experimental research design using one control and two 
treatment groups showed students in the treatment cohorts scored significantly higher than the control group. 
Students also perceived the productive failure activities as more valuable than lectures. 


Motivation and Objective of the Study 


Foundational courses in pharmacy doctorate programs help students build strong competencies to allow 
them to “integrate knowledge foundational sciences to explain how specific drug and drug classes work” and to 
apply these knowledge “to solve therapeutic problems and advance patient care”.(Medina et al., 2013). The most 
common instructional strategy used in fundamental and applied sciences builds on a scaffolding approach that 
includes in class instructor-driven worked examples of well-structured problems followed by transfer problems set 
up as homework. While worked examples are useful for well-structured problems (Jonassen, 2011), this strategy has 
major shortcomings (Darabi, Nelson, & Palanki, 2007; van Gog et al., 2015). 

First, the expert’s logical and systematic presentation of the problem-solving process tends to prompt 
learners to focus on the procedural aspects of the problem-solving mechanics. Consequently, learners often miss 
those problem-solving steps and insights that address the conceptual integration of foundational knowledge in the 
context of target topic exemplified through the worked example. Second, because the instructor assigns transfer 
problems outside the classroom learners often miss critical on-time feedback they might need when engaging in the 
problem-solving process. Since homework is intended to be a form of the learning-by-doing problem-solving task 
(e.g.Schank, Berman, & Macpherson, 2009), the lack of on-time feedback significantly hinders the effectiveness of 
this instructional task. The consequences of these two shortcomings become even more critical for topics that build 
on each other, as the gaps in knowledge and skills grow exponentially in time. 

The major objective of the intervention presented in this study was to mitigate the above weaknesses of 
worked examples by including in the instructional process a strategy that merges worked examples, homework-like 
tasks, and active learning classroom environment (Pelley, 2014; Wolfe, 2006). Next section will provide details 
about this instructional intervention. 


Instructional Context and Gap 


Pharmaceutics I is a foundation course focused on helping first year students (P1) enrolled in the in the 
pharmacy doctorate (Pharm. D.) program to create integrative bridges between the foundational chemistry course in 
pre-pharm curriculum and clinical courses in the Pharm. D. curriculum. The main expectation of this course is that 
students will build strong analytical and problem-solving skills. The instructional process in Pharmaceutics I builds 
on lectures augmented with story problems (Jonassen, 2004, p. 10) presented as worked examples. These worked 
examples build on each other, exposing P1 students to topic-specific problem-solving strategies that evolve from 
simple to complex. During the lecture, students have the opportunity to engage in the problem-solving process as 
they solve in-class transfer problems with a complexity matching that of the worked examples presented by the 
instructor. To sustain students’ engagement with the transfer problems the instructor uses virtual clickers, 
ResponseWare from Turning Technologies® (Turning Technologies, 2017) to collect students’ answers and provide 
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on-time feedback during the lecture and for student-solved in-class worked examples. For the worked examples with 
a complexity similar to the problems included in the summative assessments, the instructor assigns transfer 
homework problems to help students prepare for the exams. 

As proven by the student performance in the course, this instructional strategy proved to be effective except 
for two major topics in the course: buffered and isotonic solutions and dispersed systems. These topics are the base 
for about 80% of the problems administered in one of the major exams during the semester as well as part of the 
problems included in the comprehensive final examination. Therefore, the impact of these two topics on students’ 
performance could be significant, especially since the Pharm. D. is a mastery program that requires at least a final 
course grade of C to pass. Therefore, the instructor decided to contact college’s instructional designer to identify 
potential solutions to address this issue. 

As part of the needs analysis phase, the instructor identified three main potential constraints that can justify 
the identified gap in students’ performance. First, students learned the basic chemistry knowledge and skills needed 
for solving the problems associated with the focal topics above discussed two to three years before this course. 
Second, the level of complexity of the problems used in Pharmaceutics I course is higher than what students used to 
solve in basic chemistry courses, the time they started to build these knowledge and skills. Finally, students often 
failed to make the required connection between the conceptual aspects of chemical reactions and the algebraic 
equations used to model them. 

These identified constraints indicated that an effective solution to the target issues should simultaneously 
address several factors. First, it needs to make students aware of their level of prior knowledge in basic chemistry, 
especially when students are behind the expected level of prior knowledge. Next, the identified solution should help 
students build a valid idea of the complexity of the problems in the course. Third, the instructor needs to provide on- 
time feedback to help student gradually master the required complexity of those problems. Finally, an effective 
solution should make students aware that they need to focus on the conceptual understanding rather than surface 
elements of the problem-solving process associated with buffered and isotonic solutions, respectively dispersed 
systems. The instructional strategies and tools used at the time of the analysis, mainly the lectures augmented with 
worked examples and virtual clickers, provided support for on-time feedback and awareness of one’s lagging prior 
knowledge. However, the most critical issues, a clear picture of problem complexity and the need to focus on 
conceptual understanding of the problem-solving process, lacked support in the existent instructional strategy. To 
address this issue, the redesign team decided to implement productive failure, a failure-based instructional strategy 
that proved to be effective in increasing students’ problem-solving skill in mathematics (Kapur, 2008, 2010, 2013). 


Instructional Intervention 


Productive failure focuses on the role errors and error acknowledgment by the learner play during problem- 
solving. The core elements of the implementation of this strategy are the intentional delay of scaffolding, that is, the 
support provided to the learners prior and during the problem-solving process (Kapur & Bielaczyc, 2012). We 
propose that the main consequence of the productive failure strategy is the exposure of learners to tasks for which 
we deny them the opportunity to work within their zone of proximal development. Vygotsky defined the zone of 
proximal development as “the distance between the actual development level as determined by independent problem 
solving and the level of potential development as determined through problem under [expert] guidance or in 
collaboration with more capable peers” (Vygotsky, 1978). One of the outcomes of offering the learner all the needed 
conditions to create a zone of proximal development is a relatively low level of anxiety that is engaging rather than 
paralyzing. 

By forcing students to solve problems that are above their actual level of problem-solving knowledge and 
skills without any instructional support, the instructor denies them the opportunity to move into their zone of 
proximal development and forcing them to fail. However if students in the course will have a wide range of 
problem-solving knowledge and skills and the instructor allows students to collaborate, then the problems to solve 
need to be above the expected zone of proximal development of the peer group to force students to fail. If this 
failure would occur in a high-stake assessment context, more likely the level of anxiety will paralyze learners, 
preventing them from learning. However, since productive failure happens in low or no-stake situations, the level of 
anxiety during the failure phase will typically be at the engaging stage. This decreased level of anxiety create an 
instructional dynamics that brings the problem difficulty level closer to learners’ potential zone of proximal 
development making this strategy productive for the learning process. The main assumption behind the effectiveness 
of productive failure strategies is that when a learner is confronted with a task that is similar to a previously failed 
one, the previous failure will be referenced and act as a strong activation of prior knowledge and skills (Kapur, 
2008). 


122 


The implementation of productive failure in Pharmaceutics I had, therefore, the potential to create 
awareness among students about the complexity of the problems in this course. The expectation was that once 
students became aware of the complexity of the problems they need to solve they will focus on the underlying 
conceptual aspects of the problem-solving process rather than searching for easy-out strategies to get the correct 
answer. The integration of productive failure strategy for the buffered and isotonic solutions section of 
Pharmaceutics I, the focus of this study, followed several steps, aligned with effective strategies for failure-based 
activities (Tawfik, Rong, & Choi, 2015). 

Step 1. A short revision of fundamental chemistry knowledge that served as a base for these topics 
followed by an worked example for a simple, one concept buffer problem to determine the level of understanding of 
the students. The instructor started the productive failure session with a simple problem on the calculation of acetic 
acid and sodium acetate required to prepare a buffer solution of desired pH value (see Figure la). The problem 
directly targeted the topics discussed in class. 


Problem - 1 

What is the molar ratio of sodium acetate to acetic acid required to 

prepare an acetate buffer of pH = 5.0? The pK, for AcOH is 4.76. Responses 
- [Salt] ; : 
f pH = pK, + log —— This solution was presented after 
[Acid] results were collected with ane 
[AcONa] virtual clickers and posted! ¢ a i J 4 
/ => 5.0= 4.76 +log 1 ai 
H 4cO) Ne ‘ H 
1 sjyg MOON _ 5.9476 004 = CONG _ 96m 3 74 Totals 
. [AcOH] [AcOH] J otals 

(a) (b) 


Figure 1. Preparing productive failure: (a) Simple revision problem; (b) Answers collected with virtual clickers; 


Students performed the calculation for this first problem and submitted the ratio (a number) using virtual 
clickers. The class response (98% correct) indicated that the students followed the application of the Henderson- 
Hasselbalch equation in performing the buffer calculations (Figure 1b). 


Step 2. Failure problem. A more complex, two-concept buffer problem was then posted for students to 
solve without any prior worked example. Students were encouraged to collaborate while solving this problem and 
the results were collected with a clicker question. As part of this problem, the instructor included a two-part question 
with this problem (Figure 2a). In the first part, students needed to calculate the concentrations of the chemicals 
provided as aqueous solutions. In the second part, they needed to use the concentrations to calculate the pH of the 
buffer. 


Responses 


Problem - 2 


A buffer solution was prepared by first mixing 50 mL of 0.2 M KH,PO, with 39.1 mL of 
0.2 M KOH and then diluting the solution to 200 mL with water. Calculate the pH of this 
solution. The pK, for KH,PO, is 7.21. 
|7 KH;PO,+ KOH > K,HPO, + H,0 S \ 1 
| The added KOH converts H,PO,- (acid) to HPO,?- (salt) ' I 
1 S ' 
aon 4 0.2 M/200 mL bel = 
=39.1mLx0. m = (50 — 39.1) mL x 0.2 M/200 mL 
1 
1 =0.0391M = 0.0109 M Delayed 
' aie) Solution 
' : Salt : [HPO2"] 
H K , + log ——— +1 £2 
, 2 Pa ST Acid] 8 HPO] : 
1 3 ' 
Uae tedog eOFP) 2 9.074:0:55'—7:76 ! 
\ 0.0109 / 
ee a ee ee a 
(a) (b) 


Figure 2. Failure problem: (a) Two-concept buffer problem; (b) Answers collected with virtual clickers; 
The instructor encouraged students to discuss the problem with their neighbors. The clicker responses 


indicated 21% correct answers (Figure 2b). The responses indicated that the students found the problem to be 
challenging. Discussions amongst themselves clearly did not produce a positive impact on the outcome. The 
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problem therefore achieved the proposed goal, to create a failure context for the buffer problem solving. 
Subsequently, the instructor discussed the steps required to analyze the information provided critically and to answer 
the question successfully. 

Step 3. Once the instructor discussed the failure problem and the expected solving process he rechecked the 
comprehension with a similar, near-transfer problem. However, instead of providing the solution concentration of 
the chemicals, as it was in the failure problem, the instructor gave students the amounts of the chemicals and the 
volume of buffer required. This tweak in the problem statement intended to test students’ deep understanding of the 
conceptual aspects of the problem solving process. Again, the students discussed the solving strategies among 
themselves and responded through the clickers. 


Problem - 3 


. e Percent Count 

Calculate the pH of a buffer prepared by dissolving 20 g of H,BO, and 
25 g of NaH,BO, in 1 L of water. pK, for H;BO,: 9.24; MW for H;BO,: ==. 9.21 19 
61.83; MW for NaH,BO,: 83.12. 1639 ar ‘ 

ee ee eS aS a ee ~ 
| pH =pK, +log neo jd ‘eval 65% 2 
I [H,BO,] Solution Delaye came 92 4.65% 3 
(H,B0,)= 222" _ _ 0324 
"Seem 7 9.27 4.65% 2 
' taH,Bo,]- 22" — - 030 I 0 2.33% 1 
I . 83.12 g/mole l 
I jag 1 10.35 2.33% 1 
' pH = 9.24 +log—— =9.21 I 

0.32 r- 10.36 2.33% 1 
®* news ewe wee weewee eee ee = = - al 
(a) (b) 


Figure 3. Near-transfer problem: (a) Modified buffer problem; (b) Answers collected with virtual clickers; 


Once the clicker answers were collected and posted the instructor had some discussions with students and 
concluded that the pH values between 9.2 and 9.27 were acceptable as they emerged from significant digit rounding 
errors. Therefore, about 54% of the class responded correctly (Figure 3b). The instructor again explained the steps 
and the critical thinking needed to address the problem and emphasized the impossibility of a few calculated pH 
values (e.g., 0 and 4). 

Step 4. The last buffer problem, a three-concept far-transfer problem, closed this productive failure cycle 
(Figure 4a). As shown in Figure 4a, in this final step the instructor challenged students with a buffer problem that 
used all the concepts discussed in the previous three steps. 


Problem - 4 


Percent Count 


Calculate the pH of a buffer solution prepared by dissolving 115.0 solid citric acid (MW = 210.14) in 500 


mL water, and then adding 5.2 g solid NaOH (MW = 40.0). The pK, for citric acid is 3.13. 2.51 34.62% 18 


os ~ 
115 s 95 2 50 
4 [Citric acid] = 3,44 Mole = 0.55 mole [NaOH] = 528 mole = 0,13 mole \ a 21.15% 1 
; 2.62 13.46% 7 


| The added NaOH will react with citric acid to generate sodium citrate. After the reaction is complete: | 2.63 == a 
2.05 »./1fo 3 
| 

[Sodium citrate] = 0.13 mole/500 ml = 0.26 mole/L= 0.26 M 


3.75 3.85% = 
[Citric acid] = (0.55 — 0.13) mole/500 mL = 0.84 mole/L= 0.84 M Delayed Solution 
4.51 3.85% 2 


[Sodium citrate] 


) 


pH = pk, + log ( 1.69 1.92% 1 


! 
! 
! 
! 
! 
! 
\ 


| 
[Citric acid] 
0.26 
\ = 3.13 + log (~~) = 3.13 - 0.51 = 2.62 : P 2.52 1.92% 1 
Fe pee Pee A ey Pe eee ee ae ee ge, 2AS 10704 1 
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Figure 4. Far-transfer problem: (a) Three-concept buffer problem; (b) Answers collected with virtual clickers; 
The results were again collected with a clicker question. Following the posting of the results and after some 


discussions with the students, the instructor concluded that the pH values between 2.50 and 2.63 were acceptable as 
they emerged from significant digit rounding errors. 
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Since the goal of administering this problem was to validate the effectiveness of the productive failure 
cycle, the instructor was pleased to note that 74% students now responded with the correct answer (Figure 4b). This 
result confirmed that the expected impact of the innovative integration of productive failure in the design of the 
active learning instructional process addressed the major goal for this intervention. That is the integration of 
productive failure reduced the potential gap in student performance in solving buffered and isotonic solutions 
problems. We will discuss in more details the actual student exam performance on this category of problems in the 
results section. 


Research Questions & Methodology 


The exploratory questions in this study were: (1) Will the use of productive failure as an active learning 
instructional strategy increase students’ exam performance?; (2) Do students perceive productive failure as more 
valuable for their learning than the more traditional lecture part in the course? 

Two cohorts of P1 PharmD students, 2015 and 2016, served as treatment groups for the productive failure 
intervention, while 2014 cohort served as the control group in a quasi-experimental research design. Cohorts had 
between 81 and 85 students and while instructional tasks were mandatory, the participation in this study was 
voluntary. We used the results of an entry knowledge test administered by the instructor during the first week of the 
semester to analyze the homogeneity of the groups at the entry point in the instructional process across semesters. 
The second exam that mainly covered the target type of problems, buffered and isotonic solutions, served as the 
main measure of student performance used to test the impact of the productive failure strategy. Finally, only for the 
last treatment cohort, fall 2016, we administered a perceptional construct, adapted from the published literature 
(Grasman & Cernusca, 2015), which measured students’ perceived value of productive failure on their learning 
(Table 1). 


Table 1. Perceived Learning Value of Productive Failure 


During the Buffered and Isotonic Solutions lectures, your instructor asked you to solve a couple of difficult 
problems before discussing similar ones in the lecture. Then, once you provided your answer with 
ResponseWare (virtual clicker), the instructor explained the solution to the problem. This strategy, called 
Productive Failure, pushed you to fail in a secure instructional environment and learn from you failure. Please 
think about those lectures and indicate how strongly you disagree or agree with the following statements. 


The use of Productive Failure with ResponseWare (virtual clickers) in Strongly Strongly 
PSCI 368 helped me to: Disagree Agree 
... better retain the material taught in Buffered Solutions lectures 1 : 9 
... better prepare for Exam 2 1 : 9 
... develop a better understanding of the concepts introduced in 1 : 9 
Buffered Solutions lectures 
... feel more confident in my ability to learn the material for 1 : 9 
Buffered Solutions 


The perceptional survey was administered online during the last week of the semester using Qualtrics®. 
The perceptional construct showed a strong internal reliability with Cronbach’s Alpha = .97, being above .70, the 
accepted value for a strong internal reliability. 


Results 


While the mean values of pretest scores decreased each year (see Figure 5), a one-way ANOVA showed no 
statistically significant differences between the three groups’ mean value of prior knowledge, F (2,235) = 2.56, p= 
0.08. Tukey’s HSD test did not indicate any statistically significant differences between paired groups. 
Consequently, the three groups were homogenous at the entry point in the instructional process. 

When students’ performance on Exam 2, buffered solutions, was analyzed, a one-way ANOVA revealed a 
significant effect for the semester, F (2,247) = 25.5, p < .001. Mean values for the three groups are presented in 
Figure 5.Tukey’s HSD test showed that students in the 2016 cohort scored significantly higher on the Exam 2, 
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buffering solutions, than both 2015 treatment cohort (p < .001) and the 2014 control cohort (p < .001). The 2016 
treatment cohort also scored significantly higher than 2015 treatment cohort (p < .01) did. 


90 [%] 
80 
70 
60 


50 
40.65 


38.97 
a0 35.18 


30 


20 


10 


FS14 FS15 FS16 


M@BExam2 —Entry Knowledge 


Figure 5. Mean score values for the control cohort (Fall 2014) and respectively productive failure treatment cohorts 
(fall 2015 and fall 2016) for both the entry knowledge and Exam 2, buffered solutions. 


When we analyzed the exit survey results, regarding the perceived value of productive failure for the Fall 
2016 group, the mean value of the five items was 7.3, on an evaluation scale of | (strongly disagree) to 9 (strongly 
agree). Considering a value of “5”, the middle of the scale, a neutral opinion that will equate the value of the 
productive failure with the other instructional tasks during the lecture, we used a one-sample t-Test to test the 
significance of the mean value for the fall 2016 group. The one-sample t-Test indicated that students perceived the 
productive failure activities as statistically significant more valuable than other instructional tasks during the 
lectures, t (71) = 11.2, p< 0.001. 


Discussions 


Instructors often face rather asymmetrical results within the same course, even if they carefully designed 
and implemented what they assumed to be an effective instructional strategy. In Pharmaceutics I, the course that 
was the context for this innovation, the instructor found that the active learning strategy worked for about two-thirds 
of the course failed to produce same results for one section, the buffered and isotonic solutions and dispersed 
systems. More important, this issue proved to persist across semesters despite instructor’s continuous efforts to 
improve the instructional process by adding extra worked examples and trying to maximize the use of virtual clicker 
during the lectures. This outcome convinced the instructor to work with an instructional designer to bring an 
external perspective in the efforts to bring students’ performance for the somewhat troublesome section of the 
course. The success of the solution totally depended on the instructor’s willingness to implement it in the course and 
implement strategies to monitor the impact of that implementation. This aspect is critical since in for students 
heavily focused on high achievement, as is the case with Pharm. D. students, the implementation of the proposed 
failure-based strategy could easily fire back by stimulating students’ resistance to potential academic failure. 
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Form students’ perspective, the benefits of the proposed strategy, integration of productive failure into 
active learning classroom environment, is trifold. First, the productive failure cycles bring to the classroom problems 
with a level of difficulty that is typical for homework or exam preparation while adding a level of engagement in the 
problem-solving process that is hard to stimulate and control outside the classroom. As a result, after a cycle of 
productive failure, students leave the classroom with a level of problem-solving skills that allows them to be 
successful in solving additional transfer problems when assigned as homework or as preparation materials for an 
examination. Second, during the productive failure cycle students get multiple feedback opportunities at the time 
they need it the more, that is, as they are working on building and improving their problem-solving skills. Finally, 
the use of technology-driven instructional tools such as virtual clickers encourages students’ participation in the 
problem-solving process, as their input of the solution for the problems used in the classroom is anonymous. The 
simultaneity of these three benefits resulted from the implementation of the proposed strategy provide a strong 
support for its innovative character. 

Both by the increases in students’ performance as well as their perception of the value of productive failure 
in their learning process support the effectiveness of this strategy. As shown in the results section, students’ 
performance on the exam that previously lagged in terms of students’ performance continuously increased while 
despite the fact that students’ entry-level knowledge had a decreasing tendency. The fact that the second semester of 
productive failure implementation produced a mean performance that was statistically significantly higher than the 
first semester of its implementation also showed that this strategy requires time for the instructor to master the 
novelty of this strategy to the point to ensure its full integration into the instructional process. 

The main adjustment in the application of productive failure from one semester to the next one was the 
complexity of the problems used in the classroom. That is, in the first semester the instructor tested the strategy to, 
first, get a sense of its potential negative impact due to its “failure” part and, second, to make sure this strategy has a 
measurable impact on students’ performance on the second exam that covered the target topics, the major objective 
of this effort. The expectation is that, as the instructor will implement the strategy in the future semesters students’ 
performance for this section of the course will level out, mapping students’ performance in the other assessments in 
the course. 


Conclusions 


While productive failure is a proven effective strategy for mathematics, this is, based on our knowledge, a 
first for Pharmaceutical Sciences in pharmacy education. What makes this implementation even more special is the 
integration of productive failure in an existing active learning environment without any major disruption to the 
instructional process but with significant improvement on students’ performance. The implementation of the 
productive failure strategy in the Pharmacology course confirmed its effectiveness when correctly transferred to this 
domain in pharmacy education. Further research will focus on the potential impact of this failure-based strategy for 
other topics in the course. More important, failure-based strategies have the potential to produce positive outcomes 
in other pharmaceutics and in the pharmacy practice courses that are driven by case-based instructional tasks. 
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Abstract 


Recognizing students’ sentiment in learning state plays an important part in improving teaching and 
supporting educational decision. The construction of sentiment lexicon will help to analyze the emotional state in the 
opinion text. In this study, we focus on constructing of multidimensional sentiment lexicon automatically in the 
domain of education. We proposed a Word2vec based approach to extract sentiment words, and automatic 
categorization of these sentiment words. Experiments conducted on the review message, from online learning 
system, show that our approach leads to a better performance. 


Introduce 


Sentiment analysis refers to the task of obtaining a point of view, evaluation, attitude, emotion, etc from the 
opinion text (Liu & Zhang, 2012; Zhang & Liu, 2016). It is important to understand the sentiment orientation of 
students to predict the thoughts and behaviors of individual and groups of students, and to make educational 
decision support accordingly. Such as by analyzing the emotional expression of the students during the online 
learning process, we can more accurately predict the student's online learning course completion and more 
convenient for teachers to provide effective academic achievement prediction. Sentiment analysis has been widely 
used in a range of applications since 2001(Pang & Lee, 2008). It has been widely used in business, public opinion 
analysis, online stores and other fields (Feldman, 2013). In the domain of education, the sentiment analysis is also 
getting more and more attention from researchers. With the development of online learning, the learner's opinion 
expressed on the content of learning is an important way of communication, which often contains a lot of 
information, such as learners ' sentiment orientation towards learning content, learners ' cognitive level, the learner's 
suggestion of learning content, etc. 

Sentiment lexicon is the most fundamental unit of sentiment analysis, and most affective analysis tasks are 
based on sentiment lexicon (Pang & Lee, 2008). Sentiment analysis based on sentiment lexicon is according to 
sentiment orientation of the words provided by the sentiment lexicon, so as to analyze the text emotion under 
different granularity. The method based on sentiment lexicon mainly includes the extraction of sentiment words, the 
polarity distribution of affective words and the construction of affective lexicon. Because sentiment lexicon has 
domain uniqueness, the quality of sentiment lexicon in different fields has a great influence on the accuracy of 
sentiment analysis. Therefor researchers have constructed a domain-special sentiment lexicon , such as the sentiment 
lexicon for the sentiment analysis of Twitter(Tang, Wei, Qin, Zhou, & Liu, 2014), and the sentiment lexicon used 
for medical sentiment analysis(Goeuriot et al., 2012). However, in the domain of Chinese education did not build the 
domain related sentiment lexicon. In the Chinese sentiment lexicon, the more commonly used sentiment lexicon has 
HowNet (Dong, 2015), National Taiwan University Sentiment Dictionary (NTUSD)(Ku, Liang, & Chen, 2010) and 
Dalian University of Technology sentiment lexicon ontology library (DUTIR)(Xu, Lin, Pan, Ren, & Chen, 2008). 
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HowNet and NTUSD are not classified according to sentiment classification of emotional vocabulary, DUTIR 
although the sentiment lexicon for the seven categories, but it contains the vocabulary in the field of education is not 
commonly used. 


Purpose of study 


The purpose of this study is to acquire sentiment words with multidimensional classification in the domain 
of education. According to Ekman's six categories of emotional classification(Ekman & Friesen, 1975), we divide 
sentiment words into surprise, fear, disgust, anger, happiness, and sadness six categories. Until now, most Chinese 
general sentiment lexicon have divided the polarity of sentiment words into positive and negative. Such as HowNet 
and NTUSD, which are classified according to the simple dichotomy of positive and negative emotions. The two 
lexicons ignore the richness and diversity of human emotions, and the new emotion based on the theory of emotional 
structure Dictionaries need to break down more emotional types or dimensions. Even the DUTRI lexicon contain 
Chinese sentiment words are divided into seven categories, but most of sentiment words in it not apply to the 
domain of education. 

The emotional states contained in an educational commentary sentence is not only positive and negative in 
both states, it contains more emotional information, such as a positive comment that may contain happiness or 
surprise state. Therefore we mining more emotional information in the review text, based on the constructive multi- 
dimensional sentiment lexicon. These emotional information can accurately describe the student's learning status, 
and it can help teachers adjust the teaching plan. In addition, it can be better to assist teachers to carry out 
personalized teaching. But this is a long-term work, In order to build a more comprehensive classification of 
emotional information in the domain of education, we need to collect the text of the review in each scene of the 
teaching process. In this paper, we collected a number of educational information with rich emotional information, 
based on the data to construct multidimensional sentiment lexicon to verify the feasibility of our method. 


Method 
Using word vector 


The rise of machine learning brings new ways to natural language processing. Using machine learning to 
deal with natural language must be natural language symbolization, one-hot representation is the most commonly 
used method. This method expresses each word as a very long vector whose dimension represents the size of the 
vocabulary. The vast majority of the dimension in the one-hot representation is 0, and only one dimension is 1. This 
dimension is the current word. There's a flaw in this representation: The arbitrary words represented by this method 
are independent of each other, and there is no correlation between the two words on the word vector. Hinton first 
proposed a distributed word vector representation(Hinton, 1986), which is widely used in Deep Learning. This 
method of distributed word vector representation uses words as a low-dimensional real vector. The advantage of this 
method is that semantic similar words are closer in distance and can reflect the correlation between different words, 
thus indicating the dependency between words. Mikolov et al.(Mikolov, Sutskever, Chen, Corrado, & Dean, 
2013)Proposed CBOW model and Skip-gram model on the basis of Word Representation. The common advantage 
of these two models is that they can get high quality word vectors from scale data and better describe the 
correlations between different words.(Mikolov, Chen, Corrado, & Dean, 2013). In this paper, we use these two 
models to get the context-related word vector. 


Seed words selection 


American researcher Ekman& Friesen (Ekman & Friesen, 1975) found that humans have six basic 
emotions that are surprise, fear, disgust, anger, happiness, and sadness. These six basic emotions can be combined 
with each other into other emotions. Xu et al(Xu et al., 2008). On the basis of Ekman's research, the emotion of 
happiness is divided into happiness and good. Therefore, Xu et al. construct the sentiment lexicon contained 7 
categories of sentiment lexicon. This article will use Ekman emotional classification method, the sentiment words 
classification six categories. The seed word is selected from DUTRI. Because many of the sentiment words in the 
DUTRI are not used in the domain of education, we have chosen strong polar sentiment words as the seed words. 

Multi-dimensional classification of sentiment words based on word vector corpus and existing six 
classification general sentiment lexicon 

The basic sentiment lexicon generally divides the sentiment words into positive and negative. In the 
English dictionary WordNet-A ffect Word Word-based extended sentiment vocabulary will be divided emotions into 
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happiness, sadness, anger and fear of four basic emotions(Strapparava & Valitutti, 2004). In this paper, we propose a 
Multi-dimensional classification of Chinese sentiment words, and extract sentiment words using seed words 
containing six kinds of emotions such as surprise, fear, disgust, anger, happiness, and sadness. The extraction 
process of the sentiment words is shown in Figure 1. Word2vec can train the vocabulary of the opinion text into a 
word vector with context semantics, and by calculating the word vector of the words. If two word vectors with 
greater cosine, they have higher semantic similarity. In this study, the words with the highest similarity to the seed 
words are the sentiment words that we have to be emotionally classified. 


Seed words : : 
category 7 : Review sets 


Similarity calculation between 
seed word and the word in 
review sets 


v 


Extracted the most similarly 
words from review sets as 
sentiment words 


Od 


Automatic classification of 
sentiment words 


Figure 1. The extraction process of the sentiment words 


In the previous step, we use the seed words to extract the sentiment words with similar emotional 
categories to the seed words. But these sentiment words may be repeated in other emotional categories of 
vocabulary. For example, "4[i #2" means Very bad, it may exit in ‘sadness’ and ‘disgust’ kinds of emotions at the 
same time. In this paper, we propose an automatic classification algorithm to determine the categories of sentiment 


words, and these sentiment words are unique in each category. The automatic classification algorithm for sentiment 
words is shown in algorithm 1: 
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Algorithm 1 Sentiment words classfication 


INPUT: sentiment words(SW), Similarity between sentiment word and seed word(SIM) 
OUTPUT: sentiment words classfication:Wsu,W s,Wa,Wa,Wh, Wsa 
1: function WoRDSCLASSFICATION(SW, SIM) 


2: for each W € SW do 

3: Wi, We, W3, Wa, Ws,We «+ 0 

4: if W exist in multiple sentiment categories then 

5: W, «+ SIMy or Wo «+ SIMy or W3 <+ SIMy or Wa «+ SIMy or Ws 
SIMwy or We + SIMy 

6: end if 

a: while MAX (W1, We, W3, Wa, Ws, We) do 

8: W add in (Wsu or Wy or Wg or Wa or Wr or Wea) 

9: end while 


10: end for 
11: end function 


In the algorithm, the emotional category of sentiment word W - fear, disgust, anger, happiness, and sadness 
are expressed by Wsu, Wf, Wd, Wa, Wh, Wsa respectively. We use wl ... w6 to denote the similarity of between the 
sentiment word W and the six kinds of seed words. We use the emotional category corresponding to the maximum 
Wi (i= 1, 2, 3, 4, 5, 6) to represent the emotional category of W. For example, if W1 is the maximum, the emotional 
category of W is ‘surprise’. 

Data collection 


In the domain of Chinese education, there is no general data set for the extraction of sentiment words. In 
this study, we obtained review message from the online education platforms Mooc college 
(http://mooc.guokr.com)and Tencent class (https://ke.qq.com), which is very popular in China. We have 
crawled 8096 review message on both platforms, which mainly expresses the evaluation of the teaching content of a 
course. These reviews contain both positive and negative content, so these comments contain many words with 
sentiment orientation. 

In the course of the experiment, we used the general sentiment lexicon DUTIR, NTUSD, and HowNet. Part 
of sentiment words from DUTIR as seed words in our experiment. In the experiment, we compared the DUTIR in 
the various types of sentiment words in the frequency of comments. We found that the categories of 'anger' 
sentiment words in our collection of comments data almost did not appear. So in the experiment we did not produce 
an sentiment words associated with ‘anger’. In our research, we selected a total of 328 seed words. 


Experiments and results 


In this selection, we will introduce the parameters used in the experiment. Then we introduce the extraction 
process and classification result of sentiment words. We first use the word2vec tool, all the words in the comment 
represented by the vector. The parameters in the vector training process are set as follows: Vector size = 150, min 
_count = 5, Window size=5, alpha = 0.05, Negative Sampling= CBOW. 

The experiment is based on our proposed method to automatically extract the sentiment words and realize 
the multidimensional classification, we achieve a total of 1847 sentiment words. These words are divided into 
sadness (contain 114 sentiment words), disgust (contain 404 sentiment words), surprise (contain 26 sentiment 
words), fear (contain 58 sentiment words), and happiness (contain 1245 sentiment words) 5 categories. In order to 
verify the automatic selection and classification of sentiment words, we use artificial annotation method to verify the 
accuracy of these emotional words. The results of the manual verification are shown in Table 1| below: 
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Table I. Accuracy of sentiment words multidimensional classification 
category sadness disgust surprise fear happiness 


The 114 404 26 58 1245 
quantity 

of 

sentiment 

words 

Correct 90 300 22 40 1130 
quantity 


Accuracy 0.789 0.743 0.846 0.690 0.907 
rate 


Table 1 shows that the accuracy rate of the method is very high in the recognition of the 'happiness' of the 
category of emotion: 0.857. The accuracy of other emotional categories is also well demonstrated. The experimental 
results show that the method we proposed can be used to automatically extract sentiment words and classify the 
multidimensional degree in the text. 

Compared with the general sentiment lexicon, the sentiment words in the domain of education are only a 
small number of words exist in the general sentiment lexicon. The proportion of sentiment words in the general 
emotional dictionary is shown in table 2 . 


Table 2. Compared with the general sentiment lexicon 


General DUTRI NTUSD HOWNET 
sentiment 

lexicon 

Coexistence 409 367 385 
quantity 


Proportion 0.221 0.200 0.208 


It can be seen from Table 2 that there are only a small amount of words in the domain of education and the 
general sentiment, that is, most of the sentiment words contained in the general sentiment lexicon do not exist in the 
domain of education. In the domain of education, many sentiment words are not exit in the general sentiment 
lexicon, such as "Wr AS {#i' and "Wr {J l#', their meaning is ' not understand ' and ' understand ', they represent a 
negative or positive emotional state. The sentiment lexicon in the domain of education will be more effective in 
expressing the emotional state contained in the opinion text of education. 


Application case 


Sentiment analysis based on emotional dictionary can effectively measure the sentiment orientation of the 
text. In this paper, we build a visualization of the opinion text in the sentiment analysis platform. For example, 
students express their opinions through a comment window or an instant communication window in an online 
learning course. Teachers can perform affective analysis tasks according to the opinion text in the back end. Then 
teachers can see the result as picture 2, picture 3 and picture 4: Opinion text Visualization 
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Figure 2. Opinion text Visualization 
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Figure 3. Statistical tables of sentiment analysis 
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Figure 4. Statistical tables of sentiment analysis 


The results of the sentiment analysis visualization can be more intuitive to see the student's learning state. 
Teachers can adjust the teaching plan according to the statistical results, and stimulate students' interest in learning. 


Conclusion and discussion 


In this paper, we focus on constructing of multidimensional sentiment lexicon automatically. The lexicon 
makes up for the lack of sentiment lexicon in the domain of Chinese education. Experiments show that our proposed 
method can quickly and effectively extract sentiment words from a large number of educational domain opinion text, 
and our method effectively achieves multi-dimensional classification of sentiment lexicon. It is significance to 
construct multi-dimensional sentiment lexicon to analyze the students’ online learning opinion text. 

The words form multi-dimensional sentiment lexicon can be more detailed description of the performance 
of the text of the emotions. The emotional analysis of the educational emotion dictionary based on multi- 
dimensional sentiment lexicon has a more effect on the recognition of learners' opinion text which contained the 
learning motivation, the learning needs and the learning needs. However, there are still many improvements in the 
process of automatically building sentiment lexicon. In the experimental process, the opinion text used to construct 
sentiment lexicon is small and the sample range of corpus is limited. The corpus of the experiment is limited only to 
the environment of online learning, and the corpus of the traditional classroom is not included. In addition, there are 
some improvements in the parameters of word vector training in experiments. Word2vec is born in the English 
world, and English and Chinese have a greater difference, Therefore, there is room for optimization in the training of 
the parameters of Chinese corpus. In future work, we will continue to optimize the sentiment lexicon in the domain 
of education and apply it to sentiment analysis. 


Reference 


Dong, Q. (2015). Hownet and the Computation of Meaning. 1. 

Ekman, P., & Friesen, W. V. (1975). Unmasking the face: A guide to recognizing emotions from facial clues: 
Prentice-Hall. 

Feldman, R. (2013). Techniques and applications for sentiment analysis: ACM. 

Goeuriot, L., Na, J. C., Kyaing, W. Y. M., Khoo, C., Chang, Y. K., Theng, Y. L., & Kim, J. J. (2012). Sentiment 
lexicons for health-related opinion mining. Paper presented at the Proceedings of the 2nd ACM SIGHIT 
International Health Informatics Symposium. 

Hinton, G. E. (1986). Learning distributed representations of concepts. 


135 


Ku, L. W., Liang, Y. T., & Chen, H. H. (2010). Opinion Extraction, Summarization and Tracking in News and Blog 
Corpora. In (AAAI-CAAW. 

Liu, B., & Zhang, L. (2012). A Survey of Opinion Mining and Sentiment Analysis: Springer US. 

Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector 
Space. Computer Science. 

Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed Representations of Words and 
Phrases and their Compositionality. Advances in neural information processing systems, 26, 3111-3119. 

Pang, B., & Lee, L. (2008). Opinion Mining and Sentiment Analysis. Foundations & Trends in Information 
Retrieval, 2(1), 459-526. 

Strapparava, C., & Valitutti, A. (2004). WordNet Affect: an Affective Extension of WordNet. Paper presented at the 
LREC. 

Tang, D., Wei, F., Qin, B., Zhou, M., & Liu, T. (2014). Building Large-Scale Twitter-Specific Sentiment Lexicon: A 
Representation Learning Approach. Paper presented at the COLING. 

Xu, L., Lin, H., Pan, Y., Ren, H., & Chen, J. (2008). Constructing the affective lexicon ontology. Journal of the 
China Society for Scientific and Technical Information, 27(2), 180-185. 

Zhang, L., & Liu, B. (2016). Sentiment Analysis and Opinion Mining. Synthesis Lectures on Human Language 
Technologies, 30(1), 152-153. 


136 


Blended Instruction by Using Simulation Method Teaching to 
Enhance Digital Literacy for Student Teachers in Thailand 


Sumalee Chuachai 
Srinakharinwirot University 


Introduction 


Now, at the turn of the new century, Web technologies are replacing TV, telephones and 
newspapers as the primary means by which we are informed and entertained. (Gilster, 1997) 


The development of technology has impacted the digital interaction of people. The digital environment has 
placed a great deal of importance on the ability to successfully access and use the hardware and software necessary 
for online participation (Thompson, Jaeger, Taylor, Subramaniam, & Bertot, 2014). The ability to deal effectively 
with digital tools is often labeled digital literacy. 

Twenty years ago, Paul Gilster claimed that digital literacy was the “ability to understand and use 
information in multiple formats from a wide range of sources when it is presented via computer” (Gilster, 1997, p. 
1). However, there are a variety of definitions for digital literacy: 


Digital literacy refers to the skill, knowledge and understanding required to use new 
technology and media to create and share meaning. Digital literacy also refers to the 
knowledge of how particular communication technologies affect the meaning they 
convey, and the ability to analyze and evaluate the knowledge available on the web. 
(Hague & Payton, 2011, p. 1) 


Digital literacy involves knowing how to use a range of technology to find 
information, solve problems or complete tasks. Digital literacy is also about 
knowing how to act safely and respectfully online. (Australian Government, 2008, p. 


1) 


Spires and Bartlett (2012) classified digital literacy into three categories: (1) locating and consuming digital 
content; using strategies effectively to search for information and evaluate its accuracy and relevancy (Leu, Coiro, 
Castek, Hartman, & Reinking, 2008); (2) creating digital content; and (3) communicating digital content. Moreover, 
the Department of Broadband Communication and Digital Economy (2009) defines three core skill sets in a digital 
literate society:1) the technical ability to engage at a basic level on the computer and internet; 2) the ability to 
understand and critically evaluate digital media and digital media content; and 3) the ability to create content and 
communication. It can be concluded for this research study that digital literacy is a set of skills to locate, evaluate, 
use, create, and communicate content or information via networks in multiple media forms. 

Digital literacy has evolved into an essential literacy because the literacy of the future rests on the ability to 
decode and construct meaning from one’s constantly changing environment. The capacity to evaluate information 
on the issues of credibility and reliability is essential, as is the ability to decide when and how to apply information 
to solve problems (Spires, & Bartlett., 2012). 

There is a survival approach to developing digital literacy. Knowledge and understanding are important to 
learning. Building on the knowledge about how to use search engines and having a general knowledge of resources 
available on the Web is a critical beginning step (Moraveji, Morris, Morris, Czerwinski, & Riche, 2011). The next 
steps include: practice locating; evaluate using, creating, and communicating information in various was by 
spending more time facilitating student learning. Less time lecturing and allowing students to create or consume 
digital content may assist developing needed skills for a technological society. In addition, using mobile devices 
such as cellphones and tablets provide convenience and immediacy to the communication process for students 
(Hiller A. Spires and Bartlett., 2012). Enhancing digital literacy for students should focus on blended technology, 
especially digital technology in the classroom. 

“Blended Learning are classes where a portion of the traditional face-to-face instruction is replaced by web- 
based online learning about 30%-70% depended on class, discipline and learning objectives” (University of Central 
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Florida & American Association of State Colleges and Univers, n,d,, p. 1). Blended learning can be implemented in 
many ways, such as: 

1) students will attend face-to-face classes, study online, and do group work on a fixed schedule 

2) students will attend face-to-face classes on a fixed schedule, but they can study online by computer 
laptop that flexible schedule. 

3) students can manage their schedule to study in a face-to-face classroom and online class by themselves. 

4) students will study via an online class at home before coming to a face-to-face classroom at school, and 

5) students can take an online class with an online teacher in a computer lab, in addition to face-to-face 
classroom to provide students with more flexibility over their schedules (Clayton Christensen Institute, 2017) . 

One of benefits of blended leaning is it allow students and faculty to take advantage of much of the 
flexibility and convenience of an online course while they can get benefits of the face-to-face classroom experience. 
(University of Central Florida & American Association of State Colleges and Univers, n.d.) In addition, students 
who attend a blended course can improve social online skill while they study and practice via the online course 
because they have to create and communicate information in the online class to realize meaningful learning 
(Garrison & Vaughan, 2008). 

Furthermore, digital literacy is a construct of many skills, such as critical thinking for making informed 
judgements about retrieved information, verifying the information validity, and checking the completeness of 
internet sources (Bawden, 2008) and communicating skills to present created content via online. Thus, the 
development of digital literacy should enhance the new set of these skills at the same time. 

Khammani (2010) developed a teaching method that enhanced several skills: interaction skills, 
communication skills, problem solving skills, and thinking skill supported by a simulation teaching method. While 
Ruksakaew (2009) found that students being taught by using the online learning interaction activities in simulation 
had higher scores in interpersonal communication skills than those of students being taught without online learning 
interaction activities with simulation at a .05 level of significance. This is study it meant the simulation teaching 
model promoted the use of critical and evaluative thinking because it encouraged students to contemplate the 
implications of a scenario. In addition. it helped students understand the concepts being taught by doing an activity 
ina scenario (The University of New South Wales, 2015). This model was determined to enhance the digital 
literacy of student teachers. 

Simulation is a form of experiential learning in the imitation of real-world scenarios. Students are able to 
develop skills and experience safely (Glover, 2014). There were 4 steps in this model; 

1) Introducing simulation scenario. 

2) Defining player and role in simulation 

3) Playing in simulation and 

4) Discussion and Conclusion. (Jones, 1982; Mack, 2009; and Khammani, 2010). 

These are the reasons the researcher for this study selected blended learning and digital literacy to enhance digital 
literacy for Thai student teachers. 

In 2011, UNESSCO redeveloped the ICT competency standards it had first developed for teachers in 2008. 
As stated in the standards: professional teachers should have digital literacy and be able to: 

a) Use authoring environment or tools to design online materials, 

b) demonstrate the use of an authoring environment or tools, 

c) have participants work in groups to design an online unit, 

d) Use the network to support student coloration within and beyond the classroom, etc. (UNESCO, 2011, 
cited in NSW Education Standards Authority, 2017, p. 1). 

The NSW Government adopted all the recommendations of Great Teaching, Inspired Learning: A 
Blueprint for Action in 2013. In 2015 it was also decided to review the teacher preparation programs as the national 
priority for the Information and Communication Technology area. Graduated teachers of Australia have to 
understand: their subject area, curriculum content, and teaching strategies. In addition, they have to know how to 
integrate contemporary technologies, including digital technologies, into their teaching to enhance the learning of 
students. They also apply their ICT or digital skills in enhancing digital literacy of students (NSW Education 
Standards Authority, 2017). This blueprint for preparing teachers of NSW and the required competence levels for 
Thai teachers are similar. Thai competences were defined by The Teacher’ Council of Thailand. It indicated that 
Thai teachers have the ability to uses technology and to apply technology to student learning. 

As mentioned above, digital literacy is very important for Thai student teachers. Especially, elementary 
teachers because students in the third grade start developing digital literacy skills (Chall, 1996). For this reason, the 
researcher was interested in enhancing digital literacy of Thai student teachers who study to teach in the elementary 
education field by developing an instructional model based on blended learning combined with the simulation 
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teaching method. To design the instructional model, the first step was to set the goal of the instruction followed by 
analyzing the students to plan activities for them (Dick, Carey, & Carey, 2009). This research was the step to 
analyze students by using a survey. The result of this research will help the researcher to develop a teaching model 
to enhancing digital literacy of student teachers as recommended future research. 


Research Objectives 


The objective of this research was to survey the opinion of student teachers in Thailand on the Blended 
Instructional Model using simulation to enhance digital literacy for student teachers in Thailand. 


Research Methods 
This study is an exploratory research study with sampling methods, research tools, and procedures as follows: 
Participants 


The research used stratified sampling to randomly select 5 University of Thailand institutions that have 
Elementary Education Departments. Participants recruited for this study were 241 Thai student teachers who study 
in the elementary education filed. 

Sample size calculation was determined by using Yamane's formula of sample size with an error 5% and 
with a confidence coefficient of 95%. Multi-stage Sampling was used by classifying the Students into 5 years 
including Year 1-5 in each year were selected by using Systematic Random Sampling. Therefore, there were 277 
questionnaires distributed and 241 were returned which was equaled to 87.72%. 


Instruments 


Instruments employed in this research included a questionnaire exploring the opinion of student teachers in 
Thailand on the Blended Instructional Model using simulation to enhance digital literacy. This particular 
questionnaire consisted of 3 sections as follows; 

1) Participants’ information, 

2 Participants’ use of digital tools and 

3) Opinion of participants on the Blended Instructional Model using simulation to enhance digital literacy. 

In ensuring the technical adequacy of the research instruments, 3 experts reviewed the instruments. Those 
experts consisted of 1 expert in digital literacy, 1 expert in blended learning and | expert in simulation. They 
evaluated all the content and construct validity, and scopes and relevancy of the items. 


Data Collection 


The Researcher sent letters to the Elementary Department, Faculty of Education by post to ask for 
permission to collect data, along with an exploratory letter to certify that data obtained from the questionnaire would 
remain confidential, and the questionnaire. After the Elementary Department granted permission, the Department 
distributed questionnaires to student teachers who met the criteria identified in the questionnaire. Participants were 
asked to complete the questionnaire. Then, the Department sent the questionnaire back to the researcher by mail. 


Research Results 


The participant pool was 241 Thai student teachers all studied in the elementary education filed: students in 
first year 24.1%, students in second year 25.7%, students in third year 16.2%, students in fourth year 20.3% and 
students in fifth year 13.7%. The researcher found that all of the Thai student teachers use smart phone while some 
of the participants did not use computer or tablet (3.7%). Most of them use smart phone and computer 3-4 hours per 
day (33.1% using smart phone and 38.6% using computer/tablet) 
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Table 1. Using Digital Tools by Student Teachers 


Time/day Smart Phone Computer/tablet 
frequency percent frequency percent 

None 0 0 9 3.7 
Less than | hr.. 1 0.4 53 22.0 
1-2 hr. 23 9.6 93 38.6 

3-4 hr. 79 33.1 56 23.2 

5-6 hr. 70 29.3 23 9.5 

More than 7 hr. 66 27.6 7 2.9 
total 241 100.0 241 100.0 


The study indicates that Thai student teachers tend to use smart phones for communicating in social media 
(21.4%) more than other uses while most of them use a computer/tablet to do student papers (23.5%). In addition, 
online searching was the secondary computer use (19.6%) while communication via applications was the secondary 
smart phone use (15.6%) 


Table 2. Smart Phone Use by Student Teachers. 


topic frequency percent 
Talking on phone 122 13.3 
Communication via the applications 143 15.6 
Communication in social media: Facebook 196 21.4 
Online searching 129 14.1 
Sending or receiving E-mail 52 5.7 
Game playing 85 9.3 
Reading/Listening online news 80 8.7 
Listening to music 107 11.7 
Other 1 0.1 
total 915 100.0 


Table 3. Computer/Tablet Use by Student Teachers 


topic frequency percent 
Communication via the applications 54 7.8 
Communication in social media: Facebook 89 12.9 
Online searching 135 19.6 
Sending or receiving E-mail 68 9.9 
Game playing 45 6.5 
Reading/Listening online news 38 3:5 
Listening to music 95 13.8 
Doing student paper 162 23.5 
Other 2 3 
total 688 100.0 


140 


The findings based on the opinions of Thai student teachers regarding blended instructional by using the 
simulation method teaching were: Thai student teachers prefer simulation scenario being introduced via online but 
complete the activity in a face-to-face classroom. In addition, the most suggested advice from participants (28.2%) 
about online tools for introducing stimulation scenarios is a document in PDF form that allows students to download 
the file to read it. 


Table 4. The Findings of Opinions by Student Teacher On Blended Learning 
Online learning Face-to-Face classroom 


frequency percent frequency percent 


Step 170 70.5 71 28.5 


— 


Introducing simulation scenario. 

Step 2 111 46.1 130 53.9 
Defining player and role in the simulation. 

Step 3 104 43.2 137 56.8 


LoS) 


Playing in simulation. 
Step 4 95 39.4 146 60.6 


Discussion and Conclusion 


Table 5. Online Tools in Step 1: Introduce Simulation Situation 


frequency percent 

A document in PDF form. 68 28.2 
Content on website 46 19.1 
A thread on Facebook 22 9.1 
A Video clip on website 22 9.1 
Combine a video clip and a tread on Facebook 38 15.8 
Interaction via online real time 

Skype 46 26.6 

You Tube live 22 12.7 

Hang out video 22, 12:7 

Facebook live 48 27.7 

Line Group call 35 20.0 

Other 0 0 
Other 0 0 
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When analyzing responses by a student’s year in the program, the research found: 
1. All Thai student teachers said that the step | introducing a simulation scenario should study via online. 
2. Only Thai student teachers in the first year said the step 2 defining player and role in the simulation and 
step 3 playing in simulation should study via online. 
3. All of students said the step 4 discussion and conclusion should study in face-to-face classroom. 


Table 6. Result of Opinions by Student Teacher Year 


Step 1 Step 2 Step 3 Step 4 
Introducing simulation Defining player and _ Playing in simulation Discussion and 
Year scenario. role in the simulation. conclusion 
Online Face-to-Face Online Face-to-Face Online __ Face-to-Face Online Face-to-Face 
learning classroom learning classroom learning classroom learning classroom 
Year 44 14 30 28 31 27 24 34 
1 (75.9%) (24.1%) (51.7%) (48.3%) (53.4%) (46.6%) (41.4%) (58.6%) 
Year 44 17 27 35 27 35 20 40 
2 (72.1%) (27.9%) (43.5%) (56.5%) (43.5%) (56.5%) (33.3%) (66.7%) 
Year 25 14 15 24 8 31 12 27 
3 (64.1%) (35.9%) (38.5%) (61.5%) (20.5%) (79.5%) (30.8%) (69.2%) 
Year 31 18 24 25 26 23 23 26 
4 (63.3%) (36.7%) (49.0%) (51.0%) (53.1%) (46.9%) (46.9%) (53.1%) 
Year 25 8 15 18 12 21 14 19 


5 (75.8%) (24.2%) (45.5%) (54.5%) (36.4%) (63.6%) (42.4%) —- (57.6%) 


Discussion 


Digital literacy is one skill needed in the present and for the future. Developing digital literacy in 
elementary students is very important. Therefore, improving digital literacy of Thai student teachers is also 
important. There are several ways to enhance digital skills: blended learning by simulation was selected for this 
research. Because blended learning allows learner communication via digital tools, it helps learners develop online 
communication skills. The simulation method can enhance thinking skills, especially critical thinking; an essential 
digital skill. To achieve the goal, it was necessary to analyze the background and opinion of student teachers about 
the blended instruction model by using simulation method. This research found: 

1) All Thai student teachers use smartphones, but some do not use a computer/tablet. Smartphones are 
very functional: sending messages (SMS), calling, chatting, openning documents, checking-email, 
internet browsing and socialization (Alson & Msiagal, 2016). Moreover, nowadays smartphones are 
small and can be carried easier than a tablet. Another advantage is that the average prices of 
smartphones are cheaper than computers/tablets. 

2) Most Thai student teachers use smartphones for socialization such as Facebook. This finding is the 
same as the results of researchers Alson and Msiagal (2016). They found socialization is the most 
widely used feature on smartphones. Because there are many socialization applications for 
smartphones, it helps people to easily connect and socialize online. For this reason, smartphones are 
the most popular digital tools for socialization. 

3) Blended learning by simulation method includes 4 steps. Thai student teachers prefer the simulation 
scenario that was introduced via online (step 1) but defining player and role in simulation (step 2), 
playing in simulation (step 3), and discussion and conclusion (step 4) are activities that should be 
done in a face-to-face classroom. The results revealed in this study is that Thai student teachers still 
believe in face-to-face classroom more than online learning. 
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4) When analyzed separately by year, the results indicated that first year students preferred online 
learning. This may be because first year students are the new generation or Gen Z; born after 1977 
(FAIRVIEW CAPTAL, 2017). Technology has permeated Gen Z in their lives in many ways: they 
use cloud computing for homework assignments or make online friends around the world on social 
networks. In addition, for Gen Z, the real world and the virtual world naturally overlap (FAIRVIEW 
CAPTAL, 2017). The Researcher believhis is the reason that Thai student teachers that were first 
years prefer to study online more the others. 
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Extensive use of technology related to rapid advancements have changed the daily habits and behaviors of 
people (King et al., 2013). For instance, people have started to check their smartphones as the first thing to do in the 
morning and last thing to do at night before going to sleep. So, we have several new, technology related phobias 
such as nomophobia (King et al., 2013; Yildirim & Correia, 2015), cyberphobia (Lucas, 2015), and many more. 
The main point here is not how many times people check their smartphones or smart devices, why do they check 
their devices. According to Yildirim and Correia (2015), people afraid of losing communication and connection with 
other people at most. 

According to O’Keeffe, Clarke-Pearson, and Council on Communications and Media (2011) using social 
networking sites has become the most common activity of teens’ and adolescents’ daily life. They take pictures to 
share on “Instagram”, twit about their moods and ideas on “Twitter”, check-in a place to announce where they are 
on “Swarm”, chats with many groups on “WhatsApp”, or share videos, single photos or albums, tag their friends on 
“Facebook”. According to PEW Research Center’s (2017) social media update 2016, 79% of online adults, which 
is 68% of all Americans, use Facebook, whereas 32% of online adults, which is 28% of all adults, use Instagram. 
Moreover, according to Facebook (2017), there were 1.79 billion monthly active users and 1.18 billion daily active 
users as of September 2016. While 1.66 billion monthly active users logged in via mobile devices, 1.09 billion 
monthly active users logged in via mobile devices. When the social media usage statistics were compared it is seen 
that Facebook is the most commonly used social networking site among all social networking sites. 

The extensive use of social networking sites has already gain the attention of educators that many different 
studies have conducted to examine educational use of social networking sites, especially educational use of 
Facebook (Arteaga Sanchez, Cortijo, & Javed, 2014; Demirbilek, 2015; Henderson, Finger, & Selwyn, 2016; Manca 
& Ranieri, 2016; O’Keeffe et al., 2011; Roblyer, McDaniel, Webb, Herman, & Witty, 2010). 

Although there are many studies conducted to investigate how educators and students use social media, or 
would like to use it with or without educational purposes, this study aims to investigate how pre-service ICT 
Teachers use social media, and what do they expect from professors about social media use. In Turkey, Department 
of Computer Education and Instructional Technology (CEIT) is responsible for educating ICT teachers. The 
department has B.S., M.S., and Ph.D. degrees. People with B.S. degree in CEIT could be ICT teachers in private and 
public schools. These ICT teachers have several roles in schools besides teaching such as being a role model to both 
other teachers and students about the correct and effective use of technology, maintaining computer labs, solving 
network, hardware and software problems, and so more. Furthermore, ICT teachers are responsible for preparing 
web sites, and managing social media accounts of the institution. 

Since ICT teachers have a key role for technology integration and effective technology use in schools, they 
needed to be correctly educated. Moreover, the proper use of social media is required to be taught. At this point, 
understanding how pre-service ICT teachers use social media and what do they expect from professors might help to 
establish a road map to social media education. 

This study investigate how pre-service ICT teachers use social media, and what do they expect from 
professors in terms of using social media in education. For this purpose, a descriptive questionnaire consisted of 44 
questions, nine questions to understand social network use, and 35 questions to understand their expectations from 
professors and institutions, was prepared and utilized. At first, the questionnaire was sent to five experts with Ph.D. 
in order to ensure there was no problem. The final form of the printed questionnaire was distributed to pre-service 
ICT teachers in a public university located in central Anatolia region, and 88 responses were collected. 
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According to preliminary analysis, there were 44 (51.2%) women and 42 (48.8%) men, and their ages 
ranged between 18 to 24. The main device used to connect to the Internet is smartphones (n=60). The most common 
social network was Facebook (n=83), and least common was LinkedIn (n=22). The other social networks used by 
students were Twitter (n=45), Instagram (n=72), Google Plus (n=48), Swarm (n=42). They use social networks to 
communicate with friends (n=83), to find old friends (n=51), to find new friends (n=28), to learn what other people 
do (n=43), to share (n=60), to follow celebrities (n=44), to follow institutions (n=42), to follow news (n=74), to buy 
something (n=36). When they are asked to clarify which information they share on social networks, it is seen that 
they share their names (n=77), phone numbers (n=12), addresses (n=12), e-mail (n=30), marital status (n=21), photo 
and video (n=60), hometown (n=69), political view (n=20). 

When pre-service teachers asked to whom do they send friend request, it was seen that they send friend 
request to relatives (n=65), friends (n=72), classmates (n=69), schoolmates (n=62), dorm mates (n=57), friends’ 
friends (n=20), professors (n=47), and everybody they know (n=26). Furthermore, they accept friendship request of 
their relatives (n=68), friends (n=78), classmates (n=78), schoolmates (n=72), dorm mates (n=65), friends’ friends 
(n=41), professors (n=55), and all requests (n=13). 

These are just the initial findings. More detailed findings on what do they expect from professors and 
institutions will be presented and discussed. 
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